Rhetorical layer

As a part of the GeM model, the rhetorical layer aims to “identify the particular functional contributions made by the elements of a document to the intended communicative purposes of that document as a whole” (Bateman 2008, p. 144), that is, how the verbal and visual elements combine into a meaningful whole in a layout. The rhetorical layer draws heavily on Rhetorical Structure Theory (hereafter RST), which is a theory of text structure and organisation developed by Mann and Thompson (1987, 1988). Although Chapter 2 presented criticism of the application of linguistic concepts to multimodal analysis, particularly in connection with the metafunctions, these issues were largely related to the overextension of linguistic concepts. RST, in turn, contributes directly to only one aspect of the GeM model, and its analytic capability emerges largely in connection with the other layers of description.

Since its introduction, RST has generated interest as a theory of text organisa-tion and coherence within several fields of study (see Taboada and Mann 2006a).

Mann et al. (1992, p. 43) make the following hypothesis about text organisation, which explains the principles behind RST:

Texts consist of functionally significant parts; the parts are elements of patterns in which parts are combined to create larger parts and whole texts.

What RST does, then, is this: the theory addresses the relations that hold be-tween sequential parts of a text, which combine into larger parts and entire texts

(Taboada and Mann 2006b, p. 425). These patterns have been theorised in SFL using the concept of metaredundancy, which postulates the existence of redundant patterns across the multiple strata of language (Martin 1991, 1999). It is worth mentioning here that the possible points of contact between SFL and RST have also been explored in the work of Mann and Matthiessen (1990) and Bateman and Rondhuis (1997). While the exploration of this issue does not fall within the scope of this dissertation, it is definitely an issue that warrants further investigation, especially in the field of SFL-inspired multimodal research.

For current purposes, the goals of both RST and multimodal analysis need to be compared. Theories of multimodality put forward that the text and its func-tionality are also shaped by the contribution of non-linguistic semiotic modes (see e.g. Kaltenbacher 2004; Jewitt 2009a), which together provide — to paraphrase St¨ockl (2004) — the ‘building blocks’ of a multimodal artefact. In this connection, it should be noted that RST was used to describe the structure and coherence in multimodal documents — that is, how the parts of their structure relate to each other — at an early stage of multimodal research (see Andr´e and Rist 1995; Re-ichenberger et al. 1996). Outside the GeM framework, Taboada and Habel (2013) have recently explored the application of RST in the description of multimodal artefacts.

Bateman (2008, pp. 144-145) argues that the current models of image-text interaction in a multimodal artefact have been limited in their scope, thus neglect-ing the description of how the larger parts of the artefact work together (see e.g.

Martinec and Salway 2005; Kong 2006). As a solution to the challenge presented by the description of the overall structure of a multimodal artefact, Bateman pro-poses an application of RST. Recently, the application of RST has been criticised by Martinec (2013), who argues that RST is not semiotic but a cognitive model, because it cannot explicate the produced analyses, that is, the resulting discourse interpretations.

Martinec (2013, p. 150), however, fails to account for the fact that discourse semantics are arguably semiotic by nature. They are socially shared conceptions of discourse interpretation that have arisen over time as a result of interaction (for discourse semantics, see Section 3.3.3). And to uncover the signalling of these discourse semantics, many other analytical tools may be brought in to comple-ment the performed analyses (cf. e.g. Asher and Lascarides 2003; Bateman 2011;

Bateman and Schmidt 2012).

Considering the work undertaken here, it is obvious that the dissertation re-quires an analytical reach that covers the entire artefact. Therefore, the following discussion focuses on “classical RST” (Taboada and Mann 2006b, p. 426) in Sec-tion 3.2.3.1, and the applicaSec-tion of RST as a part of the GeM model in SecSec-tion 3.2.3.2.

3.2.3.1 Classical RST

According to Taboada and Mann (2006b, p. 426), classical RST, as it is generally understood, refers to the initial work of Mann and Thompson (1987, 1988), which defined a basic set of 24 rhetorical relations, which was later expanded to 30 relations in Mann (2005). This section presents the essential aspects of RST and its method in the following points.

Firstly, classical RST does not define a minimal unit of analysis, but as Mann and Thompson (1988, p. 248) point out that in their analysis, the defined units are essentiallyclauses. However, as Section 3.2.1 established, the minimal units of analysis in this dissertation are orthographic sentences and sentence fragments — the analysis does not extend below to the rank of a clause.

Secondly, classical RST defines relations between two non-overlapping text spans, referred to asnucleus andsatellite (Mann and Thompson 1987, p. 4). This means the text spans take on certain roles in a rhetorical relation; the nucleus car-ries the basic information, whereas the satellite contributes additional information (Taboada and Mann 2006b, pp. 426-427). However, in certain rhetorical relations, both text spans may act as the nucleus, resulting in a relation with multiple nuclei.

To account for alternative nucleus-satellite configurations, classical RST uses five schemas to define the structural relations between text spans (Mann and Thomp-son 1988, p. 247). The implications of different schema to GeM RST are discussed in the following section.

Thirdly, RST includes definitions for the rhetorical relations used in the model, which describe the relations holding between the text spans. A relation definition is made up of four fields (Mann and Thompson 1987, p. 245), which are exemplified here using the relationship of purpose³ (Mann and Thompson 1987, p. 276):

1. Constraints on the nucleus (N): presents an activity.

2. Constraints on the satellite (S): presents a situation which is unrealised.

3. Constraints on the combination of nucleus and satellite: S presents a situa-tion to be realised using the activity in N.

4. The effect: reader recognises that the activity in N is initiated in order to realise S.

A discussion of each relation defined in Mann and Thompson (1987, 1988) and Mann (2005) is not possible as a part of this thesis, owing to the focus and lim-ited space (for the definitions, see Appendix B). However, the basic principles of RST analysis should be considered here: the analyst evaluates the intentions of

3The convention of typing the relation names in small capitals, which is common in RST, is also followed in this dissertation. This is not to be confused with the earlier use of small capitals for systems in SFL.

the writer and the desired effect of the chosen rhetorical relation on the reader.

As Mann and Thompson (1988, p. 246) acknowledge, such judgements are un-certain, and therefore the statements regarding rhetorical relations areplausibility judgements. Although the analytical reliability of classical RST has not been ex-tensively studied, Taboada and Mann (2006b, p. 444) report on high analytical consistencies and agreement in several studies that involve modifications of RST and multiple analysts.

To conclude, RST appears to be a formidable theory of text organisation and coherence, which has been applied across a number of different fields of study (for an overview, see Taboada and Mann 2006a). Thus the question is, how effi-ciently RST may be applied to describe the organisation of multimodal artefacts?

Several challenges emerge after a brief consideration. For instance, organising principles of the individual semiotic modes are different. Whereas language relies on its sequential organisation, multiple semiotic modes often take advantage of the available two-dimensional space (see Bateman 2009b, 2011). In addition, the nuclearity in image-text relations seems to be highly context-dependent (see e.g.

Martinec and Salway 2005; Caple and Bednarek 2010) and ambiguous (Taboada and Habel 2013, p. 85). For these reasons, developing a consistent schema for image-text relations appears difficult. These issues and other aspects of GeM RST are considered in greater detail below.

3.2.3.2 Extended GeM RST

Bateman (2008, pp. 158-159) highlights several challenges related to GeM RST, which are described in the following points.

The first challenge of extending RST to multimodal analysis arises from its original use for text analysis. As Bateman points out, “conventional RST builds on the sequentiality of text segments” (2008, p. 157). As I argued above, all multimodal artefacts are not organised sequentially. There have been proposals of linear descriptions of multimodality in the form of potential “reading paths” (van Leeuwen 1993, 2005b), but because these semiotic models of reader behaviour remain underdeveloped, the application of the reading paths in this dissertation has to be ruled out (see e.g. Holsanova and Holmqvist 2006; Hiippala 2012c).

Therefore, alternative means of constraining the application of RST schema need to be defined.

In order to constrain the application of RST, Bateman (2008, p. 158) proposes that RST relations are restricted to hold between segmentsadjacent to each other in any direction.⁴ The adjacency of segments is defined using the area model (see Section 3.2.2.2). According to Bateman (2008, p. 158), acknowledging the

4In GeM RST, the base units that contribute to the rhetorical structure are referred to as segments.

spatiality as the principle of organisation does justice to its role in the multimodal artefacts. The spatial principle has also found support in eye-tracking studies (cf.

Holsanova and Nord 2010, pp. 95-96). Yet RST relations may also extend across layout areas in certain contexts, resulting in what Thomas (2009b, pp. 315-316) has termed “long-distance” relationships in the study of packaging design. It is possible that the tourist brochures will also include long-distance relationships, as their rhetorical structure appears to be increasing in complexity in certain cases (cf. Hiippala 2013). I will thus restrict the relations to adjacent segments, but if the analysis reveals long-distance relationships, they will be accounted for accordingly.

The second challenge relates to the application of RST schema between verbal and visual segments. In total, RST provides five schema for describing relations be-tween segments (Mann and Thompson 1988, p. 247). The two major schema types to be discussed here are nucleus-satellite and nucleus-nucleus. The nucleus-satellite schema is referred to as asymmetric (mononuclear) and the nucleus-nucleus as symmetric (multinuclear). The problem of nuclearity — deciding on schema and whether the verbal or visual element is the nucleus — was previously identified, because the image-text relations appear to be context-dependent and resistant to abstractions (see also Martinec 2013).

Although image-text relations have been extensively explored in both semiotics (Barthes 1977, 1981) and multimodal research (Royce 1998; Martinec and Salway 2005; Kong 2006; O’Halloran 2008b; Liu and O’Halloran 2009), sufficiently reliable and empirically tested frameworks have not yet emerged. What need to be avoided, then, are arbitrary assignments of nuclearity in the image-text relations. Thus, the multinuclear relationship of restatement is preferred over forced decisions on nuclearity (Bateman 2008, p. 159).

Finally, the third challenge is related to the image-text combinations on a page. Bateman (2008, pp. 160-161) identifies several cases where the classical RST is not capable of describing the observed phenomena, because the approach lacks the necessary analytical reach. Bateman (2008, pp. 161, 167) supports his argument with several examples in which image and text fragments (from an RST perspective) are connected by lines, but RST does not recognise these fragments as proper rhetorical segments. Instead of representing the relation verbally, these image-text combinations use the two-dimensional space to express the relation.

According to the previously introduced principle of spatiality, these relations need to be accounted for in the analysis. For this kind of subnuclear elaboration, GeM RST includes five additional relations based on Halliday and Matthiessen (2004) to extend its analytical reach: identification, class-ascription, property-ascription, possession and location (Bateman 2008, p. 162).

This concludes the discussion of the rhetorical layer. In the following section, I shall proceed to describe the final layer to be discussed, that is, the navigation layer.