I have had quite a few people ask me recently for key texts to get them started on multimodality, so I decided that it’s high time to share a few favourites. This list is neither exhaustive nor comprehensive – and please use the comments to post works I’ve missed out. I have taken two big eliminating decisions with the list: to only list books (though there are plenty of edited volumes here), and only those works in English. (Perhaps another day I’ll add a top ten of journal articles, but most of my favourite shorter pieces are chapters in the edited collections mentioned here. In the meantime, check through the contents pages of the journal Visual Communication, since that often contains articles on multimodality). It is also centred on Kress, Van Leeuwen and those working in their wake because that’s the path I took into multimodality – but there are other thinkers out there as well.



First though: if you read nothing else on the topic, then read these three. It was very hard picking just three, believe me.

1. Jewitt, Carey, ed. 2009 The Routledge Handbook of Multimodal Analysis. London: Routledge.

Why? It’s a go-to guide for a selection of articles on multimodality in a variety of disciplines. If you’re not sure whether multimodality will work for you, start your process of discovery here to see how broad the spectrum can be. I will never forget the article on gay dating sites, or on facebook. The introduction is also my absolute go-to for an overview of the topic.

2. Kress, Gunther, and Van Leeuven, Theo, 2006 (2nd ed.). Reading Images: The Grammar of Visual Design. New York: Routledge.

Why? Well, the 1996 first edition was the one that really started the whole thing off in the mainstream – academic and (to some extent) non-academic. It’s clearly written, it has examples from a wide range of images (a broad term which encompasses all visual ‘texts’), and it has easy-to-follow and easy-to-remember techniques which really do work for the majority of images. The particular genius of this book – and of multimodality in general – lies in identifying trends that hold true across times and cultures (and in specifying where and why they don’t, e.g. in cultures that write/read right to left).

3. Kress, Gunther and Van Leeuwen, Theo, 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.

Why? I like the way this book divides the process of a multimodal analysis into discourse, design, production, and distribution. Of course that is not the only way, and there is more to think about than those four things, but it is an excellent starting point. Yes, that is still the way I do my analyses – or at least how I begin them.


The not-quite-top three

4. Kress, Gunther, 2010. Multimodality: A Social Semiotic Approach to Communication. London: Routledge Falmer.

Why? A narrow miss of the top three, this one. It contains the best definition of ‘mode’ that I’ve seen (spoiler alert – it’s hard to define). More importantly, it unlocked for me the ‘secrets’ of social semiotics, a concept of which, until reading this, I harboured an inexplicable fear.

5. Norris, Sigrid, ed., 2012. Multimodality in Practice: Investigating Theory-in-practice-through-methodology. (New York: Routledge)

Why? This was one of the first texts I read on multimodality, and I was hooked. What could an exploding billboard possibly have in common with my work on medieval manuscripts? (Paul White’s contribution is on the billboard; I found striking resonances to my own work.) Carey Jewitt’s introductory chapter is another of her clear, helpful outlines of the topic. My only quibble with this volume in the title – surely they could have thought of something easier to remember (and to capitalise correctly!).

6. O’Hallaran, Kay L. and Smith, Bradley A., eds, 2011. Multimodal Studies: Exploring Issues and Domains. New York: Routledge.

Why? This is another broad-ranging collection of essays. More than that, reading them created echoes in my mind. Watch this space for a blog post on manuscript marginalia based on Van Leeuwen’s assessment of patterns in this volume. Lim, Nekmat, and Nahar offer a striking take on multimodality and media literacy. And there are other, equally exciting chapters here, including theoretical.


And finally: four specialist works

7. Jewitt, Carey, and Kress, Gunther, 2003. Multimodal Literacy. New York: Peter Lang.

Why? Education, teaching, and learning is one of the major players in multimodality. This book is an excellent introduction to the topic. If I have a criticism of it, that would be its focus on children. Of course much is also applicable to adult learners. There is a further reading list on multimodal literacy to be found at the project website here.

8. Page, Ruth, ed., 2010. New Perspectives on Narrative and Multimodality. Routledge: New York.

Why? Two main reasons. First, the mode of ‘written language’ is too easily dismissed (and I myself am guilty of doing so). Of course, ‘written language’ is a mode made up of many other modes, and this book gets into some of these. Second, there are contributions which cover opera, Safran Foer, metaphor, gesture, and more. I love a wide-ranging yet coherent and inspiring collection (as this list shows).

9. Van Leeuwen, Theo, 2008. Discourse and Practice: New Tools for Critical Discourse Analysis. New York: Oxford University Press.

Why? I know this list is a little Van Leeuwen heavy, but he deserves it. Like the Page volume, this one is helpful for getting into the nitty gritty of the multimodality of written language. However, it does so from a very different viewpoint – that of linguistics. I have always been scared of linguistics, yet I understood and even enjoyed this book. So I can’t help but recommend it.

10. Van Leeuwen, Theo, 1999. Speech, Music, Sound. Basingstoke: Macmillan.

Why? This is to music what Reading Images is to visual studies. In general, music is still somewhat under-represented in multimodal studies (though things are improving there, and not just because of me, obviously). Read this and be inspired.


But don’t forget…

There are works which didn’t make this list for not being strictly on ‘multimodality’, yet which deserve a mention at least. Barthes, oh Barthes, the first French philosopher I ever made sense of. Barthes speaks volumes of use to multimodalists (even when they disagree). Derrida (yes, I know, just go out and read the whole lot, eh?) has a lot to say about writing. Deleuze and Guattari’s Mille Plateux (OK, the translation, but only because my library doesn’t have the French) is permanently on my desk and travels with me just about everywhere – I can dip in and out and always find something inspiring. Remediation (Bolter & Grusin) and Moving Media Studies: Remediation Revisited (eds Philipsen and Qvortrup work as a good pair. We also mustn’t forget the grandfathers McLuhan and Halliday. And if you can read any of the Scandinavian languages, then there is a wealth of scholarship, particularly from Denmark and Norway, a tradition of which I’m now proud to be a (minor) part.

Right, reader, what have I missed that just has to be included here?

This blog post has been a long time coming as I had to write a paper for a super-exciting conference at the University of Virginia which I will be attending next week. (For details see here.) As part of this paper I revisit the familiar medieval figure of Fortune, particularly her wheel.

BnF fr. 9221 (E), fol. 206v

BnF fr. 9221 (E), fol. 206v

Throughout Machaut’s seminal Voir dit Fortune’s wheel turns, first raising then lowering the lovers’ fates. (This is argued by Jacqueline Cerquiglini-Toulet in her book Guillaume de Machaut et l’écriture au XIVe siècle (Paris: Champion, 1985), if anyone wishes to pursue it further.)

In my paper next week I will argue that the placing of the final group of Machaut items in the Pennsylvania Chansonnier (UPenn 902, viewable here) in fact reverses this wheel by re-using images from the Voir dit and representing them in the new format, often using different lyric items, and in reverse order, thus returning the lovers to their state of anticipation at the start of the tale. This whole thought process, then, is the background to this short reflection on circles and cycles.

N21 coverFor now, however, a step back (or ahead, since we are briefly leaving the Middle Ages). In 2009, the remarkable anthology Notations 21 was published by Mark Batty publishers. It is a collection of graphic scores in honour of the legendary John Cage, specifically marking the 40th anniversary of his own collection entitled ‘Notations’. (The project is ongoing: see for more details.) It is, in some ways, a chansonnier, since it contains musical compositions (and additionally works of art inspired by the music), however, all of the works are presented with their creator’s name attached. Interestingly, though, the cover does clearly state ‘By Theresa Sauer’, clearly giving her authority over the collection. Naturally, she is listed as ‘author’ in many of the catalogues in which this book appears. Yet, although being the driving force behind the collection, she is but one of many creative artists whose work it portrays. In scholarly terms, she is closer to an editor, or to a director, than to a book’s ‘author’ in the traditional sense. Yet her name, placed as the highest textual item on the cover, is a striking witness to her involvement in the collection, and not at all unlike the rubric which we saw in the last blog post here invoking Machaut’s authority over one of the manuscripts transmitting his works.

Nevertheless, in this book there is one work in particular which draws its inspiration from the form of the circle, or, perhaps more accurately, the cycle. It is entitled ‘Performing Notation, Notation Performing’, and is found on p. 147.Maxwell Performing Notation, Notation Performing 1b
Many medievalists will recognise the similarities between the visual presentation of this piece and one of those by Baude Cordier in the celebrated Chantilly codex (Musée Condé ms 564).

Both have the structure of one large circle surrounded by four smaller ones. In the Notations 21 piece the principal circle is broken, rolling off into the difference. Cordier’s work is well behaved, enclosed within its page. Both works feature music in the central circle and text in the outer circles, though the piece in Notations 21 has a lengthy prose text within its principal circle. Of the outer circles, those in the Notations 21 piece offer quotations on music – including one from John Cage himself – whereas the small circles in the Cordier piece provide instructions for performance and additional verses. Both pieces make use of a limited colour palette, and present the composer’s name: Cordier’s at the head of the piece, that of ‘Performing Notation, Notation Performing’ somewhat more modestly, yet nevertheless unashamedly, across the bottom of the page.

Both works are entirely hand drawn, one on parchment, the other on paper. Now, both are only available to view via the transcoded means of the computer screen or the book page: Notations 21 contains a high-quality reproduction, the Chantilly codex is kept under strict lock and key but is available in facsimile. Indeed, while the ‘original’ codex is indeed still in existence for the Chantilly codex, the French postal system saw to it that Notations 21 is, in fact, the ‘original’ for ‘Performing Notation, Notation Performing’, since the precious piece of paper itself never made it home.

Unlike many of the items in Notations 21, ‘Performing Notation, Notation Performing’ contains no explanation, no instructions. This was a deliberate decision, since I preferred to leave any interpretation to performers/readers. What I have said so far in this post is merely what someone with some knowledge of medieval music would be able to infer without too much forethought, and I do not wish to speak as the work’s composer. What follows, then, is written not from the point of view or the composer, but as an interpreter, looking at the work in the light of my current multimodal project. Imagine, then, if you will, a line across the page: any composer’s insider knowledge that may have been present stops here.

In the seminal Reading Images: The Grammar of Visual Design by Gunther Kress and Theo van Leeuwen (Abingdon: Routledge, 1996, 2nd ed. 2006), they offer the following analysis of the forms of the circle:

‘Circles and curved forms generally are the elements we associate with an organic and natural order, with the world of organic nature – and such mystical meanings as may be associated with them derive from this. Angularity we associate with the inorganic, crystalline world, or with the world of technology, which is a world we have made ourselves, and therefore a world we can, at least in principle, understand fully and rationally. The world of organic nature is not of our making, and will always retain an element of mystery. Curved forms are therefore the dominant choice of people who think in terms of organic growth rather than mechanical construction, in terms of what is natural rather than in terms of what is artificial.’ (p. 55)

It is not a great leap of deduction, then, to posit that the use of circles here to present visual music is a nod to music’s natural qualities, that is to say that music is a function of humankind that is not of our making and not entirely of our knowing. Both pieces bend the staves into a circular shape, thus transmorphing that which seeks to contain music to let it speak a new shape, a natural shape. In ‘Performing Notation, Notation Performing’ the broken circle can be seen both as a distortion and an escape: the music escapes from the staves and, indeed, from the page. The notation starts which neumes and moves chronologically through various versions until it can no longer be contained on paper: in the eleven centuries following the invention of musical notation in the West only a minority of music has ever been written down, and collections such as Notations 21 only go to show that the boundaries of the stave are as fragile now as they ever were.

Additionally, both pieces make extended use of the areas of ‘centre’ and ‘margin’ on the page, and Kress and van Leeuwen can help us with that, too:

‘[I]f a visual composition makes significant use of the Centre, placing one element in the middle and the other elements around it, we will refer to the central element and Centre and to the elements around it as Margins. For something to be presented as Centre means that it is presented as the nucleus of the information to which all the other elements are in some sense subservient. The Margins are these ancillary, dependent elements. In many cases the Margins are identical or at least very similar to each other, so that there is no sense of a division between Given and New and/or Ideal and Real elements among them. In other cases … Centre and Margin combine with Given and New and/or Ideal and Real.’ (p. 197. Explanations of the horizontal ‘given’ and ‘new’ and vertical ‘ideal’ and ‘real’ can be found on pp. 181 and p. 186 respectively.)

But what of the reading path of such images? ‘Performing Notation, Notation Performing’ invites a reading path whereby the more accessible text, that is the text in the margins, is read first, since this does not require turning an unwieldy book or screen (particularly a screen which ‘helpfully’ keeps images the ‘right’ way up when turned). Thus the marginal texts enter the reader’s mind first, and it is having read them – and probably the composer’s name – that the reader will turn to the words which are at the centre of both the page and importance. Finally, the reader may begin to decipher the notation, and decide whether and how the elements of the page should be rendered together, and what to do about the palindromic title.

Cordier’s ‘Tout par compas’, on the other hand, invites a reading which begins with the words under the musical notation, and then, for this reader at least, a preliminary look at the notation itself. Before proceeding too far with this challenging task (and it is challenging, even more so today than for those who grown up seeped its conventions), however, I look to the marginal texts for instructions, for clues. Once these have been assimilated, my attention returns to the musical notation. There are enough ‘clues’ provided that a coherent performance can be obtained from the manuscript; indeed, here is one now, courtesy of youtube and, more specifically, Ensemble Organum (it’s high time we heard some more music on this blog):

Tout par compas

This brief survey of two works has indeed been brief. It has not attempted a detailed analysis of either, merely drawn some links between them and offered some thoughts on circles and cycles. But one of the most intriguing things to have emerged from it, for me, is the link between both the graphic compositions and Fortune’s wheel. Look again at the miniature from BnF fr. 9221 (Machaut manuscript E) up at the top of the page. See those four circles within the wheel? Yeah, so do I. It seems that Katelijne Schiltz has had something to say about ‘Tout par compas’ and Fortune’s wheel, though I have not yet been able to consult a copy of her article ‘Visual Pictorialism in Renaissance Musical Riddles.’ in Journal of the Alamire Foundation 4:2 (2012), pp. 204–21. I very much doubt that she considers Notations 21 or, indeed, multimodality in the article, however. Nevertheless, if I can undraw that line I put in earlier and speak for a moment as the composer – I did not think of Fortune’s wheel. That is not to say that we can’t see it in there, of course. For that is one of the great delights of multimodal thinking: the onus is put on the reader to re-create the text. In other words, if you want to read ‘Performing Notation, Notation Performing’ as an allegory of Fortune, then that’s fine by me. Or, in a more scholarly working, the background of the reader of any textual being will impose itself on that reader’s re-interpretation of the text, and this does not invalidate that reading. A multimodal reading is not about uncovering hidden (or unhidden) codes sent from creator to receiver; rather, it is about the reader creating the text anew. Like the music escaping from the page in ‘Performing Notation, Notation Performing’, a multimodal interpretation is free from the constraints of author intention or, indeed, faithfulness to authority. In the context of Machaut studies especially, that is a breath of fresh air. L’auteur est mort; vive le lecteur.

This list is not exhaustive, nor is it in any particular order. If you think I should be following your blog, please contact me using the form at the bottom of the page. I’m sorry they don’t have fancy links; I haven’t yet worked out how to put them in without them sitting on the right side of my page. (That’s what you get for trying to build a website in two days.) I’m sure that, with the joys of web 2.0, someone will soon sort me out.

Medieval Song Network:


Burnable Books:

Bethany Nowviskie:

Elizabeth Eva Leach:

J.P.E. Harper-Scott:

Modern Medieval:

In the Medieval Middle:

In the Hall of Mirrors:

Jill Walker Rettberg:

There must be more… Go on, tell me!