Multimodality as a field of study
2.1 Origins and development
2.1.3 Current challenges
It may be suggested that some of the challenges facing multimodal research do not only result from the inherent multimodality of human communication and our need to understand it, but also from the broad developments currently taking place in the society, which shape the way we communicate with each other. As O’Halloran et al. (2010, p. 3) write:
The twentieth century was a time of rapid change and growth in the study and understanding of human meaning systems. Whereas scholars prior to the mid twentieth century were on the whole in the relatively safe position of being able to restrict themselves to the study of (for the most part written) language, the ongoing revolution in multimedia de-sign and digital technology within contemporary society at large has led to a proliferation of multimodal documents (using media such as graph-ics, digitized photographs, audio and video texts, contemporary digital media, in particular web-based media, and three-dimensional virtual
objects). Meanwhile, the twentieth century has seen a profusion of disciplinary and theoretical perspectives upon human communication in general and multimodal communication in particular.
The challenge is thus presented by new and increasingly complex forms of semiosis across a range of different media. As I pointed out above, multimodal research explores semiosis using linguistically-informed methods, which underlines the rapid development of disciplinary and theoretical perspectives since Barthes proclaimed that “linguistics stops at the sentence” (1977, p. 82).
According to Kong (2006, p. 209), the proliferation of communicative situa-tions and artefacts involving multimodality has lead to the point whereby existing theories have to be refined for a more accurate description of the studied phe-nomena. However, the challenge presented to multimodal analysis by information technology may also provide solutions to analytical problems within the field. In recent years, significant advances have been made in incorporating information technology in multimodal analysis: O’Halloran (2008b) has used graphics editing software to analyse artefacts in static media, while Tan (2009) has demonstrated how software can be used to identify and deconstruct complex multimodal phe-nomena involving moving images. O’Halloran et al. (2010, 2011, 2012) and Smith et al. (2011), in turn, have continued the discussion of designing software for mul-timodal analysis and its implications to the field.1 Finally, practical applications of information technology to multimodal analysis have been exemplified by the automatic visualisations described in Thomas (2009b) and Podlasov et al. (2012).
In linguistics, information technology has contributed significantly to the rise of corpus linguistics and provided the means for increased empiricism (see e.g. Biber 1988; Leech 1991). Teich (2009, p. 113) sums up the development succinctly:
“linguistic work is unthinkable today without the support of computers”. This ob-servation applies increasingly to research in multimodality as well, as corpus-driven methods have been carried over to multimodal research, for example, in the work of Baldry and Thibault (2005), Baldry (2007), Bateman (2008), Velasco (2012), Taboada and Habel (2013) and Kong (2013). Additionally, the potential applica-tions of multimodal corpora have also been examined by Thomas (2007, 2009b) and Hiippala (2012c), who have explored the possibilities of interfacing Bateman’s (2008) model with concordancers, optical character recognition and eye-tracking equipment. However, Parodi (2010, p. 72) notes that although standard markup languages such as XML may provide the backbone for multimodal corpora, work remains to be done in developing theories of multimodality that would enable the researchers to harness the full potential of corpora, especially in the area of auto-matic segmentation and annotation of data (see e.g. Allwood 2008; Flewitt et al.
1This work has later resulted in commercial applications for multimodal analysis, see multimodal-analysis.com.
2009). Future work is therefore likely to involve developing efficient means of cre-ating corpora, and improving theory-building by using corpora to test hypotheses and feeding these findings back into the theory.
One domain of multimodal research where a close relationship between theory, data and information technology is currently emerging is the analysis of film and television. In this domain, several research streams have appeared following the early multimodal investigations (see e.g. van Leeuwen 1991, 1996; Thibault 2000).
To begin with, the development and application of computer tools for multimodal analysis has been particularly strong in the aforementioned work of O’Halloran (2004a), Tan (2009) and O’Halloran et al. (2010, 2011, 2012). Another recent development has involved multimodal descriptions of the various aspects of ‘filmic’
semiosis, which also draw on film studies and document theory (Bateman 2007, 2009c; Tseng 2008, 2009; Tseng and Bateman 2010, 2012; Bateman and Schmidt 2012).
The ongoing work on the analysis of film and television has also contributed to the analysis of digital media, which allows dynamic content in the form of moving images. Digital media has also received increased attention in multimodal research, most likely due to its growing role in contemporary communication. Some examples of recent research include websites (Kok 2004; Djonov 2007; Bateman et al. 2007; Knox 2007; Hopearuoho and Ventola 2009; Tan 2010), presentation software (Djonov and van Leeuwen 2011a,b), digital art (Huemer 2008) social networking sites (Eisenlauer 2011) and online games (Maiorani 2009; Stenglin and Djonov 2010).
In this connection, it should be noted that the major theories of multimodality have emerged during a period of rapid technological change in the last 20 years.
During this period, the role of information and communication technology has grown in terms of both availability and everyday use. This development has pro-vided the multimodal analysts with a wealth of new situations and phenomena to study, but concerns about the theoretical consequences of the development have also been raised. As Bateman (2008, pp. 213-214) writes:
... a word of caution for the currently exploding ‘study of websites’
being undertaken in multimodal linguistics: there are useful and inter-esting aspects of websites to investigate, but they are found exceedingly rarely directly on the surface in terms of novel design and new genres.
To what extent the web-page has moved beyond technological depen-dence to support semiotically interesting meaning-making possibilities is an open question, a question that is not addressed adequately simply by assuming it to be the case.
Increased attention should therefore be paid to the development of empirically-oriented frameworks for the analysis of digital media. However, before advancing
to the description of dynamic digital media, we need to have a solid understanding of the basic principles of multimodal meaning-making, to which this dissertation contributes by modelling the structure of a static artefact. An understanding of the basics is also a prerequisite for applying the newly acquired knowledge of multimodality, as Livingstone (2004, p. 12) points out:
Until we have a robust account of the media in which people might be judged literate, we can say little about the nature or uses of their literacy.
In short, an assumption that a rapid change in semiosis has taken place should not be made on the basis of transition from one medium to another. Language and other semiotic resources are unlikely to change at such a rapid pace, because their users would be unable to keep up with the change. As I will show in this disser-tation, tracking the changes in the functions and structure of the semiotic modes is necessary for describing multimodal communication in the world today. By observing the past, we may better understand how the present situation emerged.
To conclude, the state-of-the-art in multimodal research may be summed up in two points. Firstly, the increasing use of information technology should be accom-panied by a simultaneous drive to develop and redefine the theoretical frameworks to a degree where the performed analyses (1) may be replicated using different data and (2) are not limited to their immediate analytical context. Secondly, research evidence from outside the field of multimodal research should not be ig-nored simply because it does not have the same underlying theoretical principles and a compatible metalanguage (see Section 2.1.2). By combining information technology and corpora with a carefully defined theoretical concepts, this disser-tation accommodates both of these points and brings them together in a model of artefact structure.
Finally, despite the increased availability of analytical and technological tools, the theories that comprise the theoretical foundation of contemporary multimodal research may have received less attention than they warrant (cf. Zhao 2010a). I argue that this is especially reflected in the systemic-functional and social semiotic theories and their extension to multimodal research, in which the same theoretical concepts are often applied to describe different semiotic resources. In any case, the contribution of social semiotics to the study of multimodality has been and continues to be remarkable, and therefore an understanding of social semiotics and the associated linguistic theories is a prerequisite for a theoretical discussion of multimodality. For this reason, I shall now move to discuss the influence of systemic-functional linguistics and social semiotics on the theories of multimodal-ity.