I have had quite a few people ask me recently for key texts to get them started on multimodality, so I decided that it’s high time to share a few favourites. This list is neither exhaustive nor comprehensive – and please use the comments to post works I’ve missed out. I have taken two big eliminating decisions with the list: to only list books (though there are plenty of edited volumes here), and only those works in English. (Perhaps another day I’ll add a top ten of journal articles, but most of my favourite shorter pieces are chapters in the edited collections mentioned here. In the meantime, check through the contents pages of the journal Visual Communication, since that often contains articles on multimodality). It is also centred on Kress, Van Leeuwen and those working in their wake because that’s the path I took into multimodality – but there are other thinkers out there as well.
First though: if you read nothing else on the topic, then read these three. It was very hard picking just three, believe me.
1. Jewitt, Carey, ed. 2009 The Routledge Handbook of Multimodal Analysis. London: Routledge.
Why? It’s a go-to guide for a selection of articles on multimodality in a variety of disciplines. If you’re not sure whether multimodality will work for you, start your process of discovery here to see how broad the spectrum can be. I will never forget the article on gay dating sites, or on facebook. The introduction is also my absolute go-to for an overview of the topic.
2. Kress, Gunther, and Van Leeuven, Theo, 2006 (2nd ed.). Reading Images: The Grammar of Visual Design. New York: Routledge.
Why? Well, the 1996 first edition was the one that really started the whole thing off in the mainstream – academic and (to some extent) non-academic. It’s clearly written, it has examples from a wide range of images (a broad term which encompasses all visual ‘texts’), and it has easy-to-follow and easy-to-remember techniques which really do work for the majority of images. The particular genius of this book – and of multimodality in general – lies in identifying trends that hold true across times and cultures (and in specifying where and why they don’t, e.g. in cultures that write/read right to left).
3. Kress, Gunther and Van Leeuwen, Theo, 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.
Why? I like the way this book divides the process of a multimodal analysis into discourse, design, production, and distribution. Of course that is not the only way, and there is more to think about than those four things, but it is an excellent starting point. Yes, that is still the way I do my analyses – or at least how I begin them.
The not-quite-top three
4. Kress, Gunther, 2010. Multimodality: A Social Semiotic Approach to Communication. London: Routledge Falmer.
Why? A narrow miss of the top three, this one. It contains the best definition of ‘mode’ that I’ve seen (spoiler alert – it’s hard to define). More importantly, it unlocked for me the ‘secrets’ of social semiotics, a concept of which, until reading this, I harboured an inexplicable fear.
5. Norris, Sigrid, ed., 2012. Multimodality in Practice: Investigating Theory-in-practice-through-methodology. (New York: Routledge)
Why? This was one of the first texts I read on multimodality, and I was hooked. What could an exploding billboard possibly have in common with my work on medieval manuscripts? (Paul White’s contribution is on the billboard; I found striking resonances to my own work.) Carey Jewitt’s introductory chapter is another of her clear, helpful outlines of the topic. My only quibble with this volume in the title – surely they could have thought of something easier to remember (and to capitalise correctly!).
6. O’Hallaran, Kay L. and Smith, Bradley A., eds, 2011. Multimodal Studies: Exploring Issues and Domains. New York: Routledge.
Why? This is another broad-ranging collection of essays. More than that, reading them created echoes in my mind. Watch this space for a blog post on manuscript marginalia based on Van Leeuwen’s assessment of patterns in this volume. Lim, Nekmat, and Nahar offer a striking take on multimodality and media literacy. And there are other, equally exciting chapters here, including theoretical.
And finally: four specialist works
7. Jewitt, Carey, and Kress, Gunther, 2003. Multimodal Literacy. New York: Peter Lang.
Why? Education, teaching, and learning is one of the major players in multimodality. This book is an excellent introduction to the topic. If I have a criticism of it, that would be its focus on children. Of course much is also applicable to adult learners. There is a further reading list on multimodal literacy to be found at the project website here.
8. Page, Ruth, ed., 2010. New Perspectives on Narrative and Multimodality. Routledge: New York.
Why? Two main reasons. First, the mode of ‘written language’ is too easily dismissed (and I myself am guilty of doing so). Of course, ‘written language’ is a mode made up of many other modes, and this book gets into some of these. Second, there are contributions which cover opera, Safran Foer, metaphor, gesture, and more. I love a wide-ranging yet coherent and inspiring collection (as this list shows).
9. Van Leeuwen, Theo, 2008. Discourse and Practice: New Tools for Critical Discourse Analysis. New York: Oxford University Press.
Why? I know this list is a little Van Leeuwen heavy, but he deserves it. Like the Page volume, this one is helpful for getting into the nitty gritty of the multimodality of written language. However, it does so from a very different viewpoint – that of linguistics. I have always been scared of linguistics, yet I understood and even enjoyed this book. So I can’t help but recommend it.
10. Van Leeuwen, Theo, 1999. Speech, Music, Sound. Basingstoke: Macmillan.
Why? This is to music what Reading Images is to visual studies. In general, music is still somewhat under-represented in multimodal studies (though things are improving there, and not just because of me, obviously). Read this and be inspired.
But don’t forget…
There are works which didn’t make this list for not being strictly on ‘multimodality’, yet which deserve a mention at least. Barthes, oh Barthes, the first French philosopher I ever made sense of. Barthes speaks volumes of use to multimodalists (even when they disagree). Derrida (yes, I know, just go out and read the whole lot, eh?) has a lot to say about writing. Deleuze and Guattari’s Mille Plateux (OK, the translation, but only because my library doesn’t have the French) is permanently on my desk and travels with me just about everywhere – I can dip in and out and always find something inspiring. Remediation (Bolter & Grusin) and Moving Media Studies: Remediation Revisited (eds Philipsen and Qvortrup work as a good pair. We also mustn’t forget the grandfathers McLuhan and Halliday. And if you can read any of the Scandinavian languages, then there is a wealth of scholarship, particularly from Denmark and Norway, a tradition of which I’m now proud to be a (minor) part.
Right, reader, what have I missed that just has to be included here?