The ‘visual turn’ in the tourist brochures

The implications of the so-called ‘visual turn’ in communication have been dis-cussed in various fields of study (see e.g. Kress 1998; Livingstone 2004; Duncum 2004; Mitchell 2005). In multimodal research, both theoretical (Iedema 2003) and methodological (Bateman et al. 2004; Parodi 2010) concerns have been raised in connection with the visual turn. Moreover, specific observations have been made about artefacts that are functionally close to the tourist brochures. As Kong (2006, p. 222) writes:

Travel guides exhibit a range of styles, from content that is extremely packed with words without any pictures to content that is fully illus-trated with images and pictures. This reflects the history of the genre

— texts produced more than 10 years ago tend to be more word-based, whereas those produced in the last 5 years are usually illustrated with more pictures and images, although the degree of integration between words and images can vary considerably.

Kong’s observation raises a question relevant for this dissertation: has there been a similar development in the tourist brochures?

To investigate this issue in the annotated corpus, I performed two queries on the occurrence of graphic elements. The first data set returned by the query, represented by the line in Figure 6.4, consisted of the base units withalt attributes, which indicate visual content (see Section 5.2.1). Drawing on the layout layer, the second set of data consisted of the total size of the graphics elements in the realisation information, which had any of the following type attributes: photo, illustration, and map. The second data set is represented by the bars in Figure 6.4. I shall now discuss both queries in greater detail.

To begin with, it should be noted that the average percentage of visual base units in the data is relatively low (17.4%). The low average can be explained by the GeM annotation schema. A photograph constitutes a single base unit in the base layer, whereas the minimal unit of analysis for language is limited to an orthographic sentence or a sentence fragment (see Section 3.2.1). If the number of visual layout units were compared to that of verbal layout units — that is, entire paragraphs — the mean value for the percentage of graphic elements would have been higher. Nevertheless, a broad survey of the distribution of visual and verbal content can already tell us many useful things about the multimodal structure of the tourist brochures, as I will show below.

Note the range of variation in the number of visual base units in Figure 6.4 and that in several instances, the brochures contain only linguistic base units.⁸

8MEH 1967;CEH 1967;WTZ 1978; WTH 1983;SMH 1982;HWY 1999;BNO 2006.

0%

1967 1967 1972 1976 1978 1983 1984 1986 1988 1999 2002 2008

Percentage of layout area occupied by graphics

Percentage of visual base units

Figure 6.4: Graphic elements in the annotated corpus

Furthermore, in the light of the proposed visual turn, these non-visual instances appear throughout the years represented in the data (1967-2008), shown on the horizontal axis in Figure 6.4. On the one hand, among the “range of styles” (Kong 2006, p. 222) for the tourist brochures, there also exists a configuration that prefers purely linguistic communication. On the other hand, Figure 6.4 also shows spikes where the percentage of visual base units exceeds 50% (1976, 2006, 2008).

This appears to represent an alternative configuration, which relies more on visual communication, as opposed to its verbal counterpart. As I will show, the spikes and dips in Figure 6.4 correspond to specific configurations of the semiotic modes in the tourist brochures, which are described later in Chapter 8.

To conclude, the base layer can only quantify some very basic properties of the tourist brochures. Among the two opposites, the spikes and the dips in Figure 6.4, there are configurations that seem to divide their communicative tasks between the verbal and visual modes without preferring neither. It is precisely here where the attention needs to be focused, if we aim to study what Kong (2006, p. 222) calls the “degree of integration” between words and images. I have previously argued that to describe the integration of the semiotic modes, it is necessary to combine information from multiple sources: this also applies to the investigation of the visual turn in print media (cf. Mitchell 2005).

Consequently, the bars in Figure 6.4 combine information from two different components of the layout layer, in order to evaluate the effect of the postulated visual turn on the layout space and its use. The information was retrieved from the area model and the realisation information components of the layout layer. This was done using a query, which first calculated the total amount of available layout area and then proceeded to calculate the size of each graphic element present in

the layout. The result shows that the graphic elements take up a significant part of the available layout space in the annotated corpus: the average amount of space taken up by graphic elements was 50.5%.⁹

As I noted above, the calculated size for the graphic elements included only the following element types: photo, illustration, and map. This means that certain two-dimensional elements, such as demarcating lines and in-line icons were ex-cluded from the count, thus focusing on the aforementioned graphic elements that are known to do communicative work (see e.g. MacEachren 1995; Machin 2004;

Boeriis and Holsanova 2012). As Figure 6.4 shows, images have been present in the tourist brochures throughout the period of time represented by the data.

Therefore, it is not surprising why it has been suggested that the graphic elements play an important role in the tourist brochures, particularly in the formation of an image of the destination (cf. Molina and Esteban 2006).

What comes to the extensively discussed visual turn, it appears that the tourist brochures have been fairly visual all along, at least according to the annotated cor-pus. It should be kept in mind, however, that the visual turn does not necessarily manifest itself as a growing proportion of graphic elements or the space they use (Hiippala 2013, p. 466). Instead, if a visual turn has taken place, it may also manifest itself as novel configurations of the semiotic modes: I will investigate this possibility in Chapter 8.

In conclusion, I want to refer back to the hypothesised model of change for multimodal artefacts and genre, shown in Figure 3.1. On the basis of the model, I propose that the postulated visual turn may be largely based on everyday observa-tions ofsemogenic processes, particularly in terms of phylogenesis — the long-term expansion and change that affects the semiotic modes (see Section 4.3.2.1). Hu-mans appear to be sensitive to phylogenesis, which enables us to estimate when a multimodal artefact was produced. This kind of phylogenetic sensitivity may be a part what has been described as “document” or “functional” literacy (see Cohen and Snowden 2008; Waller 2012), and casual statements about certain multimodal artefacts being ‘retro’, ’vintage’ or ’old-fashioned’ may reflect this sensitivity (cf.

Kress and van Leeuwen 2001, pp. 72-73).

Whether the phylogenetic sensitivity also enables us to accurately describe the specific characteristics — such as the degree of visuality in multimodal artefacts

— without analytical training is a completely different issue. I thus suggest that a more neutral perspective to the phenomenon can be achieved through multimodal corpora. As the annotated corpus shows, the tourist brochures have been relatively visual all along: this calls for a systematic investigation of the so-called visual turn,

9In some cases, the value exceeded 100%: this results from a limitation of the GeM model in measuring and annotating non-geometric shapes (see Section 5.2.2.1). These values have been cut off at 100%.

in order to avoid working with mere assumptions. It is important to raise this point, because the possibly unfounded notions of a visual turn may also influence the society at large through educational policy: I shall explicate my argument below.

A proper investigation of the visual turn should thus involve computer-based approaches and larger data sets. Again, the work of Michel et al. (2011) is a promising lead: the data includes a wealth of media that use both language and image, for instance, various types of printed magazines. After circumscribing a corpus consisting of a specific medium and retrieving the data, it would be possible calculate the distribution and proportions of verbal and visual content, as I have done in Figure 6.4 above. Turning the manual process into an automatic one, however, will require the input of specialists in computer vision (see e.g. Okun et al. 1999). Combined with appropriate methods of multimodal analysis — such as those developed in this dissertation — a model of the multimodal artefact could be used as the basis for identifying the preferred multimodal structures.

Consequently, the research described above could inform us whether (1) the communication has indeed shifted towards the visual in the studied media, and (2) if this shift has also affected the multimodal structure of the artefacts realised in the medium, thus answering the call for writing the history of “visual language”

(Kostelnick and Hassett 2003, p. 231). This kind of knowledge could prove valu-able for informed policy-making in the various fields of education (see e.g. Hobbs 1998; Freedman and Stuhr 2004) and considerably sharpen our understanding of what goes on in a multimodal artefact. This knowledge could support a consider-ation of what needs to be taught and how to reach a level of functional literacy required in the world today, encouraging questions such as:

• Do we need to teach the critical analysis of the content or should we focus on teaching how communication is structured multimodally?

• Should we emphasise the interpretation of visual content or the combinations of both verbal and visual content?

• If all aspects presented in the questions above are relevant, what is the most efficient way of teaching them simultaneously?

Providing the answers to these questions will most likely require input from mul-tiple disciplines, but most importantly, all of the answers will need to be informed by empirical research. Moreover, if we are indeed advancing into an age defined by multimodal communication, the effects of technology need to be considered as well. For this reason, in the following section I discuss the impact of technologi-cal development on the structure of a multimodal artefact. To keep the discussion within the scope of this dissertation, I shall focus exclusively on the aspects of pro-duction, as opposed to the technologically-mediated consumption of multimodal

artefacts (for research in this area, see e.g. Leander and Vasudevan 2009; White 2010, 2012; O’Halloran and Smith 2012).