Rank scale

Several theories of multimodality (see e.g. O’Toole 1994, 2011; O’Halloran 2008b) propose that language and image possess a hierarchical structure that may be de-constructed into analytical units. The deconstruction of the semiotic modes is also necessary for this dissertation, as the comparison of artefacts requires the notions of structural constituency and hierarchy. In systemic-functional linguistics, the rank scale is a theoretical construct used to describe the compositional hierarchy of language, in which the compositional layers of clause, group or phrase, word and morpheme are organised by the relation of “is a part of” (Halliday and Matthiessen 2004, p. 20).

The concept of rank scale was adapted to multimodal analysis early on by Kress and van Leeuwen (1990). A more elaborate application was later presented by O’Toole (1994), who used the concept to deconstruct various multimodal forms of art, such as painting, sculpture and architecture (for a wide range of later ap-plications, see e.g. O’Halloran 1999b; Alias 2004; Guo 2004; O’Toole 2004; Ventola 2011). Furthermore, O’Toole combined the rank scale with the metafunctional principle (see Section 2.3.3 for a discussion) into a framework for describing the functions and systems in painting (O’Toole 2011, p. 24). The rank/function matrix has been frequently used to describe the interaction and relations between language and text in various contexts (see e.g. O’Halloran 2005, 2008b; Martinec and Sal-way 2005; Guijarro and Pinar Sanz 2008), although O’Toole has later pointed out that there was no intention to present the rank scales for language and image as corresponding structures with direct mapping in both semiotic resources (Thomas 2009a, p. 54).

At this point, it should also be noted that the notion of a rank scale is not uncontested. Zhao (2010b), who problematises the application of rank scale in multimodal research, notes that the issue has been extensively debated in mul-timodal research (cf. Martinec 2005). To put it simply, the issue is whether the segmentation of multimodal data into analytical units advances the analysis or leads to “infinite detail” (cf. Forceville 2007, p. 1236). Zhao also observes that

“the flexibility in theoretical conceptualisation ... has led to confusion and low consistency in analytical practices” (2010b, p. 254). Moreover, Zhao (2010b, pp.

261-262) argues that the relationship between multimodal research and SFL has been so far largely monolateral as opposed to dialogic. In this connection, it is useful to recall the notion of SFL as an exotropic theory presented in Section 2.2.1 and the dialogic principles behind such theories. While the contribution of SFL to multimodal analysis is unarguably significant, it appears that the

short-term, ‘purpose-oriented’ borrowing can have a negative effect on the theoretical development of multimodal research in the long run.

Nevertheless, the work of O’Toole has remained influential in multimodal re-search, especially in the efforts to determine corresponding structures across semi-otic resources and their interaction within the SF-MDA framework (see O’Halloran 2004b, 2008b). In this approach, the rank scale has proven particularly strong in the analysis of film and television, where it has been used to deconstruct the complex and intensive processes of multimodal meaning-making (cf. Tan 2009; O’Halloran et al. 2010, 2011). Because the rank scale continues to be an important tool for the segmentation of multimodal data and is still being developed (Boeriis and Hol-sanova 2012), the concept and its applicability to the analysis of print media — in combination with the notion of metafunctions — will be considered again in Sec-tion 2.5.2. I will now continue with a descripSec-tion of the metafuncSec-tions, which are another central theoretical concept often deployed together with the rank scale.

2.3.3 Metafunctions

The concept of metafunction is another influential theoretical construct that has been carried over to multimodal research from SFL and social semiotics. The metafunctional principle was introduced in the early work of Halliday (1970/2002a, 1973, 1978) to simplify the process of accounting for the form and functions of language. As Halliday points out in an interview by Paul Thibault (1987, p. 607):

[T]he notion of metafunction is simply an attempt to capture this re-lationship between the internal forms of the language and its use in contexts of social action.

The three metafunctions — the ideational, the interpersonal and the textual — are fulfilled simultaneously in every instance of language use. According to Eggins (2004, p. 2), these metafunctions realise the different but interwoven “strands of meaning”, which O’Halloran (2008b, p. 444) describes in the context of multimodal meaning-making as follows:

The metafunctional principle is the principle that semiotic resources simultaneously provide the tools for constructing ideational meaning (i.e. experiential meaning and logical relations) and for enacting so-cial relations (i.e. interpersonal meaning). These metafunctions are enabled through the organisation of discourse, which is the textual metafunction of semiosis.

The quote above illustrates how the concept of metafunctions can be applied to describe the functions of a given semiotic resource. The metafunctions are regarded

as omnipresent, simultaneous and dependent on each other. As a result, no lin-guistic or multimodal ‘text’, whether written or spoken, can be purely ideational, interpersonal or textual, as the realisation of a text requires contributions of each metafunction. At the same time, each metafunction produces a different type of structural output in language (Halliday 1979/2002b, p. 200).⁶ Finally, as the metafunctional principle has already received considerable attention in linguistic research (see e.g. Halliday and Hasan 1986; Martin 1991; Jones and Ventola 2008).

I will mainly focus on the application of metafunctions in multimodal research.

The works of Kress and van Leeuwen (1996, 2006) and O’Toole (1994, 2011), which have wielded considerable influence on the development of multimodal re-search, have also extended the metafunctional principle to multimodal analysis.

Both Kress and van Leeuwen and O’Toole used the metafunctional principle in their theoretical frameworks to analyse the semiotic construal of various multi-modal artefacts or their parts, such as advertisements, photographs, paintings, architecture, sculptures, etc. Because O’Toole focused mostly on art and archi-tecture, I will pay special attention to the work of Kress and van Leeuwen and the particular research stream originating from their work, which are closer to the interests of this dissertation.

In Kress and van Leeuwen (1996, p. 183), the authors sum up the aim of their social semiotic approach to multimodal analysis:

We seek to be able to look at the whole page as an integrated text.

Our insistence on drawing comparisons between language and visual communication stems from this objective. We seek to break down the disciplinary boundaries between the study of language and the study of images, and we seek, as much as possible, to use compatible language, and compatible terminology in speaking about both, for in actual communication the two and indeed many others come together to form integrated texts.

To achieve the required integration, Kress and van Leeuwen (1996, pp. 40-42) adopted the metafunctional principle to account for the functions that visual com-munication needs to serve. In addition, the metafunctions provide a framework for contrasting the processes of verbal and visual meaning-making. For instance, Kress and van Leeuwen (1996, p. 49) illustrate how ideational meanings are in-stantiated using the linguistic system of transitivity, which is used to establish

6For a more detailed discussion of the relation between metafunction and structure, see Martin (1996, 1997). For a discussion of the issue in a multimodal context, see O’Halloran (2008b, pp.

447-448).

a relationship between a process and its participants.⁷ In image, transitivityis realised using vectors: I will present an example later in Section 2.4.2.1.

Although the influence of Kress and van Leeuwen’s approach is evident in the wealth of research following the social semiotic tradition, which also speaks for a high descriptive capability and adaptability to describe various types of data (see e.g. the following edited volumes: Baldry 2000b; O’Halloran 2004b; Ventola et al.

2004; Jones and Ventola 2008; Ventola and Guijarro 2009; Jewitt 2009c; Dreyfus et al. 2010; Bednarek and Martin 2010; Baldry and Montagna 2011), there are some concerns about the capability of the metafunctional principle to serve the needs of this dissertation. These concerns are now addressed below.

Let us begin by considering the metafunctional principle as a key heuristic for multimodal analysis. Machin (2009, p. 182) argues that the metafunctions (in combination with the rank scale) allow a more precise description of communica-tion and its multimodal aspects than the approaches that separate the analysis of different semiotic resources and their contributions. As we have seen, the concepts of metafunctionality and rank scale have proven capable of deconstructing the build-up of meaning in multimodal artefacts. While the metafunctions and rank scale may bring us closer to an understanding of how stratified semiotic systems work (cf. Halliday 1979/2002b, pp. 196-197), that is, how meanings are struc-tured on the strata of content and expression and the principles that govern these semiotic processes in multimodal contexts (cf. O’Halloran 2008b), several open questions about the limitations and capabilities of the metafunctional principle remain.

For instance, Kress and van Leeuwen (2002, p. 345) have discussed the emer-gence of the semiotic resources within a culture:

It is clear that cultures do not expend the same energy at all times on all the potentially usable semiotic resources: hence some are highly developed and become fully articulated for all the communicative and representational purposes of that group, while others are partially ar-ticulated or hardly at all.

What Kress and van Leeuwen suggest is that all semiotic resources are not neces-sarily similar in terms of the metafunctional structure. This means that although a semiotic resource may possess a metafunctional structure, certain aspects of the organisation may be underdeveloped, especially if contrasted with other semiotic resources at play. Concerns about the extended applicability of metafunctions have also been raised in O’Halloran (2008b, p. 451), who points out the need to develop

7In SFL, the names of systems are commonly written in small capitals. I will follow this convention in this chapter. In later chapters, the small capitals are used to indicate rhetorical relations.

alternative approaches to “cross-functional systems”, such as colour and typogra-phy, which operate in the expression stratum. St¨ockl (2004, p. 12) also classifies colour as a sub-mode of both language and image, implying that colour is not a fully-fledged semiotic resource (see Figure 2.3). At the same time, colour may have various communicative functions which may be captured using the metafunctional principle, suggesting that in certain contexts, colour has the meaning potential of a full-blown semiotic resource (Kress and van Leeuwen 2002; van Leeuwen 2011).

Given the uncertainty regarding the development of semiotic modes within a culture and their functions, it is necessary to consider whether the metafunctional principle is a methodologically robust starting point for deconstructing the struc-ture of a multimodal artefact. Without a prior knowledge of (1) how the semiotic resources and their functions have developed within a culture, i.e. how much en-ergy has been expended into shaping them, and (2) how these semiotic resources and their configuration may be contrasted to other artefacts and social situations, developing a framework to capture the structure of a multimodal artefact becomes a challenging task. Because the metafunctions are primarily concerned with mean-ing, their coupling with particular kinds of multimodal structure remains unclear.

Therefore, it may be suggested that without comprehensive empirical research to identify and describe the underlying systems that realise the meanings we de-scribe using the concept of metafunctions, the metafunctions are too broad for describing complex multimodal phenomena, and they are thus limited to making generic observations about multimodality. This is evident especially in the applica-tion of the metafuncapplica-tional principle to the analysis of layout (see Secapplica-tion 2.4.2.3).

For this reason, the metafunctional principle is subjected to a more critical treat-ment in Section 2.4, which discusses the social semiotic approaches to the analysis of print media.