Medieval/Digital Multimodalities

Multimodality and Medieval Multimodalities

Mark Amsler
Comparative Literature (European Languages and Literature)
School of Cultures, Languages, and Linguistics
Faculty of Arts, University of Auckland
m.amsler@auckland.ac.nz/mark.amsler@gmail.com
auckland.academia.edu./markamsler
Draft for Multimodality and Medieval Multimodalities Seminar
New Chaucer Society, London, 12 July 2016
 
Multimodality and Medieval Multimodalities
            Multimodality the concept seeks to capture how different semiotic modes and activities function in expression and communication. Semiotic modes include linguistic and nonlinguistic activities and frames (visual, oral/aural, kinetic [i.e. movement, gesture], spatial [proprioception, exteroception]; Poyatis 2002). Multimodality the analytic approach aims to describe and understand varied socio-cultural practices whereby people use linguistic and nonlinguistic modes to represent, communicate, and interact (Kress 2003; Kress and van Leeuwen 2001). A multimodal approach can be historicized, historically informed, or critically theorized. But here’s the hitch. Multimodality as praxis is not a special way of making meaning but the default setting for animate and creaturely experience in general. Like other animals and plants, we experience through sensory perceptions what we come to acknowledge and cognize as the world. Through various sensory experiences, we generate, experience, and respond to one another and the world with embodied affect and understanding. So the question presses us: What is the relation between multimodal literate, linguistic, or communicative activities and multimodal perception and understanding in general?
          When we shift our attention from the abstract noun multimodality to the concrete and plural noun multimodalities, we ground the concept in praxis. Kress and van Leeuwen (2001, 2003; Kress, “Multimodality,” in Multiliteracies 2000) rework Halliday’s model of systemic functional grammar (SFG) (1994; with Matthiessen 2014) and sketch out multimodality as the basis for complex expressive linguistic design and communicative practices which transform oral, literate, visual, and performative meaning making. Halliday distinguishes three metafunctions in human language: ideational (representation of the world), interpersonal (communication, interaction), and textual (relevance of context). He develops his scheme based on a social semiotic view of communication, which compares favourably with Jakobson’s well known six semiotic modes model (1960). Although Kress moves outside the linguistic field and discusses TV, cinema, and Web-based texts as contemporary multimodal media, his approach remains grounded in Hallidayan linguistics. Taking a different approach than Halliday, Scollon and Scollon (2003) foreground multimodal semiotics as communicative interaction and meaning making as situated and contextual (cf. Multiliteracies 2000). Just what is a “context,” what we mean by the concept of a "context," is a controversial topic in semiotics, linguistics, and multimodal research. Here, I posit that contexts include the immediate situation and pragmatics of utterance as well as, and just as importantly, broader social, institutional, and cultural contexts. Recognising that all meaning is "situated" is just the beginning of critical praxis. Contexts include both production and intention situations and, just as importantly, reception and uptake. Although broader contexts sometimes go underrecognized in linguistic- and multimodal-oriented research or are rendered invisible through ideological naturalization or repression, they nonetheless shape and contribute to the diverse meanings participants, makers and users, assign to sensations, emotions, activities, and events. On a global scale, perhaps the most relevant context for all semiotic and performative situations of utterance is the non-appearance or denial of institutional or cultural multimodal and communicative contexts, the "zero point" of epistemology and the justification of universalising discourse (cf. Castro-Gómez 2005; Hayles 2002).
           Medieval multimodalities depend on medieval semiotics, but the relations are sometimes hard to get ahold of. The diverse medieval discussions of the sign, its nature, function, and classifications, emerged primarily within theological and philosophical discourse (especially Augustine De magistro [1954], De dialectica [1975], De trinitate, Bk. 15 [1968], De doctrina christiana, Bk. 4 [1962]) or within scholastic theory and commentaries on Aristotle, Priscian, Lombard’s Sentences, and other canonical Arts and Theology texts. If and how that intellectual discourse moved into everyday practices, social experience, or literary texts, if it does, remains unclear outside specific cases. Vance (1989), Zumthor (1973), Nichols (1983), and other critics have productively read medieval literary texts through a broad and historically informed semiotic frame. With the exception of Zumthor, these scholars aim to interpret or situate literary texts within their original manuscript and/or cultural contexts. That is, these historically informed readings of literary texts use some form of the concept of multimodality to locate literary signification within the philological or the signifying, setting Textual Studies or the once and future New Philology on par with New Literacy Studies as proposed by the New London Group and others. To use Halliday’s terms for a moment, these medieval critics focus their attention on the textual and ideational rather than the interpersonal or interactive. Le Goff (1988) suggested a broader historical view of medieval cultural formations, but much of the Annales school’s approach still relies on discerning the intersections of discrete kinds of production (literary, artistic, architectural) and social structures (especially rank, sex, and authority). Related to medieval semiotics, late medieval grammar did take account of modes (modi) which make up Grammatica, including modes of signifying and mode of affect (e.g. Pseudo-Robert Kilwardby 1975), but we need to think further about how discussions by Kilwardby, Roger Bacon (2013), Anselm of Canterbury (1998), and others compare with contemporary accounts of multimodality and Peircean semiotics.
            In what follows I take on the question of medieval multimodalities by focusing on literacy and literate practices as kinds of multimodal experience. The interrelations of words and pictures in medieval manuscript culture and literacies are fundamental. Here, I will focus more on words and pictures in relation to what I have elsewhere called “affective literacy” and which some refer to as “embodied literacy” (e.g. Hayles 2002). I shall also look at other kinds of medieval multimodal practices and understanding which combine perceptual and representational modes -- verbal + visual + movement, pictorial + verbal + nonlinguistic sound -- to produce “texts” with nondiscrete sensible, affective, and cognitive borders. Multimodality and meaning making cannot be divorced from perception and the senses. Medieval understandings of the senses and emotions were key elements of meaning making as praxis and uptake. Focusing on some specific kinds of medieval meaning making in multimodal contexts, I explore and parse how our historically informed understanding of manuscript multimodalities contributes to the larger question of multimodalities across times, spaces, and media.
            Like literacy, language acquisition then and now is fundamentally a multimodal activity, oral/aural, visual, and kinesics. We know and have known a good deal about language acquisition, and we can feed that knowledge back into our historical inquiry into multimodal understanding. I acquire a first language in immediate physical and social circumstances, in my home and in the neighbourhood, at school, at church, and around the corner. The expanding radius of acquisition develops a dialectic between PRIMARY and SECONDARY discourses, home and the wider social world, within which my language usage is structured and given agency and meaning. I also acquire a first language throughout embodied experience. I recall vividly my then two-year-old son’s first encounters with New Zealand English in daycare. One night at the dinner table sitting in his booster seat, he wanted to say something about the temperature of his food. He also wanted to use some of the new linguistic sounds he was hearing in the University’s preschool. Trying to articulate the word HOT with a New Zealand rounded back vowel (the sound in PAW, rather than FATHER), he literally surged up and forward with his body in his seat to pronounce the sound and the word which contained it. He was lifting the vowel sound with his entire small body as he uttered a word he knew with the new vocal sounds he was hearing in a new speech context.
            Augustine and other medieval language theorists observed how a child learns to speak not only by repeating what she hears linguistically but by approximating what she hears and sees and feels bodily and by taking pragmatic cues from mature speakers as to articulation, sound sequences, and gestures. Second language acquisition is no less experiential and pragmatic, but is necessarily filtered through one’s acquisition of a first language. Grammar books, written dialogues, and vocabulary lists are important for L2 acquisition, but they are not sufficient for competent speaking, listening, or attitude nor for many literate needs. No one speaks alone. Speaking in one’s skin paradoxically requires that we speak in relation to what we sense, perceive, and recognize. Learning to read and write complicates the pragmatics of language depending on the degree of multimodality in praxis.
            It is difficult to get ahold of how people after 600 CE acquired Latin or a second vernacular as a spoken and written language, but there are provocative traces and suggestive sources as to multimodal functions. Take accent or speech sounds, for instance. Following the Late Latin tradition, medieval grammarians characterized LITTERA as having name (nomen), shape (figura), and sound (potestas) (cf. Amsler 2010). This traditional triadic scheme identified and parsed writing as being made up of elementa and as having both a visual and an oral/aural mode. A person produces or reads a text by perceiving strings of letters visually with the eyes and “completing” those visual perceptions with vocal sounds from the mouth while cognizing the larger units, words and phrases, as coherent text. The oral-aural-visual process is complemented by textual markup and pragmatic markers which help the reader construe various levels of text, from individual words to paragraphs to larger sections. The medieval understanding of the phenomenon of writing included both sight and sound as fundamental sensory experiences. Medieval literacies were embodied and performative. Despite the increasing use of silent or subvocal reading strategies (cf. Saenger 1995), writing and reading involved oral, aural, visual, and embodied actions and interactions. Medieval literacy expanded the multimodal frame further by including pictorialism, hypertext markup, and diacritics.
            Following the medieval theory of LITTERA, vocal sounds (spoken, chanted, or sung) completed the inscripted text, but which sounds? Throughout the Middle Ages, people used various spellings and proposed several different character systems to take account of natural changes in articulation or to guide or control pronunciation of a text in the face of such local linguistic changes. Some Late Latin grammarians noted the growing asynchrony between received characters and current pronunciations, that is, the perceived changes and variations in Latin pronunciations in different language contact situations and over time. Sometimes these grammarians used particular letter combinations to mark language change and describe different pronunciations; other times they lamented the sound changes as the decay of “proper” spoken Latin as encoded in traditional written forms.
            In the early seventh century Boniface (c. 675-754 CE) worried about his Germanic accent when pronouncing Latin as a second language. When Pope Zacharius (675-752 CE) summoned Boniface to be examined in Rome on his orthodoxy and methods for dealing with dissent in the Church, the “Apostle of Germany” asked that he be allowed to write his responses in Latin and send them to the Pope’s advisers “so that the silent letter (muta tantum littera) alone may reveal my beliefs in a well-reasoned manner (rationabiliter)” (1916; my translation). Rather than completing the inscription, Boniface regarded his Latin speech sounds as a problem. As a nonnative Latin speaker and writer, Boniface (born in England as Wynfrid) expressed more confidence in his ability to write “reasoned” textualised Latin than to speak the southern European variety as pronounced in Rome. Boniface implies that Latin “belongs” more to southern Europeans than to northern Europeans. In his letter, Boniface identified himself as a peregrinus, foreigner, to Latin, someone whose pronunciation of L2 Latin was less prestigious than the variety spoken by natives of southern Europe or Rome. On the one hand, Boniface appealed to a restricted Latin modality, visual, grammatical, and silent rather than spoken, to articulate his beliefs and rationalisations. Written Latin perceived and cognized visually is presented as a transregional, almost transcendental norm, a shared discourse which floats above vernacular and oral situations. At the same time, Boniface assumes that the written Latin text he sends to Rome will be pronounced with a local accent. The Roman Latin accent completes the silent but seen written mode with a more persuasive and acceptable set of linguistic sounds. Despite Boniface’s claim that he can present his arguments more rationally in writing than speech, he still expects his written text to be remediated as speech through reading aloud, either individually and privately or more likely aloud before the Pope and his investigators or perhaps silently cognized with local mental speech sounds by one or more individual readers. By strategically manipulating the multimodal reception of his letter, Boniface relies on writing’s rendering of visible language to guide speech even as speech completes his written text. Releasing a visible text into the communicative situation, Boniface hoped to neutralize what he believed would be the linguistic stigmatization of the Germanic writer by the Roman readers.
             In the twelfth century, the so-called First Icelandic Grammarian (FIG) and the English priest and sermon writer Orm(in) reveal other multimodal responses to language variation. Around 1150 FIG compared with grammatical and multimodal sophistication Latin and Icelandic phonology and writing systems. He deployed the grammarians’ traditional three-part description of the letter (nomen, figura, potestas) to carefully distinguish sound and shape. FIG examined the pronunciation of Old Icelandic as he heard it and then proposed an orthographic reform: “I have used all the Latin letters that seemed to fit our language well and could be rightly pronounced, as well as some letters that seemed needful to me, while those were taken out that did not suit the sounds of our language. Some of the Latin consonants were rejected, and some new ones added. No vowels were rejected, but a good many were added, since our language has the greatest number of vowel sounds” (ed. Haugen 1972:12-13; my modified translation). Whereas Boniface sent his primary written Latin letter into a mixed field of different, hierarchically ordered accents, FIG worked to better adapt and correlate the inscription of the vernacular with the actual available sounds of the local spoken variety. Boniface and FIG order visual and oral linguistic modes differently according to whether the written text or speech is the primary mode of linguistic representation. In both cases the sounds of a language, embodied performative speech, play an important semiotic role.
            In the mid-twelfth century at Bourne Abbey, Lincolnshire, Orm (Ormin), a canon of the Augustinian order, composed the Ormulum (Bodleian Lib MS Junius 1)a set of versified vernacular homilies with paraphrases of Gospel texts and exegesis for “Ennglissh follc, for lufe off Crist …” (line 19). Orm’s allegorical exegesis is based primarily on Bede and the Glossa ordinaria. The single autograph manuscript is incomplete, uses almost no hypertextual mark up, and contains corrections in two hands, one of which is the author/original scribe’s. Despite being in verse, Orm’s text is laid out in continuous prose lines, similar to the layout of some Old English poetry manuscripts.
             In his preface, Orm expresses his dismay at how people are “mispronouncing English,” and he instructs copyists (it’s not certain there were any) to use his unique orthography to control speech and produce what he regards as proper English pronunciation, which he implies is more understandable to the local community and will support their better understanding of Christian scripture and doctrine. Orm uses rhythmic verse lines with fifteen beats (8/7) per clause line to indicate linguistic stress. The verse rhythm is based on the Latin iambic septenarius form, but I think Orm was very likely presupposing local knowledge and expecting that his clerical readers were familiar with the “fifteener” in oral vernacular verse. Orm’s linguistic localism also included his coinage or adoption of new English vocabulary to express the Biblical text (e.g. neddrestre(o)n “serpent race” [line 9265], replacing OE nædercynn). Orm uses several orthographic conventions to indicate vowel length. If a syllable ended in a consonant (closed), he doubled the final consonant to mark the preceding vowel as short. If a syllable ended in a vowel (open), he added a diacritic after the vowel to indicate length. Orm also used distinct letter forms to mark the emerging two different pronunciations of English G : the traditional OE ʒ for the two sounds [d͡ʒ] and [j] and the more recent character g for [ɡ], the latter reflecting the influence of French-language scribes.
             Orm’s metatextual orthographic features and careful spellings and corrected respellings of English words guided clerical readers to pronounce the English Gospel paraphrases and exegesis according to the perceived preferred current accent among the vernacular community.[1] In other words, Orm’s written page accessed by sight provided specific, programmatic information as to how the reader should vocalize the text to lay listeners. Presumably, the metatextual cues also enhanced clerical readers’ comprehension of the vernacular homilies, whether they read sotto voce or silently to themselves. Pragmatically and contextually, Orm’s project applied the criterion of correct Latin pronunciation in liturgy and reading scripture aloud to the oral articulation of vernacular religious texts. More than fifty years before the Fourth Lateran Council (1215), Orm presumed that English priests were responsible for the salvation and proper instruction of their lay congregations by clearly speaking aloud vernacular scriptural text. Using speech sounds cued to written characters and hypertextual markup, Orm deploys the authority of the visual literate mode, based on the primacy of the translated Latin text, to standardize and control different priests’ oral articulation of vernacular scripture and sermonic discourse. Orm’s project privileges written text with coded hypertextual signifiers and metatextual codes (characters, spellings, verse stress patterns) to signal to priests the oral quality and quantity of East Midlands English vowels which their audiences expected to hear.
            For a long time, Orm’s text has been classified as a unique source for the changing pronunciation of English in the East Midlands after the Norman Conquest. It is that. However, I think the Ormulum also suggests something about medieval multimodal literacy more generally. In Orm’s text, the goal of multimodal (oral + visual) textuality is not so much linguistic accommodation or standardization as it is using written and hypertextual markup forms to make oral communication accessible and understandable to a vernacular speaking audience. After the Norman Conquest English speech sounds within institutional and sociocultural contexts were rendered newly problematic. In effect, the Ormulum textually privileges and instantiates, perhaps for the first time in England in over one hundred years, a form of perceived vernacular pronunciation, lexicon, and grammar understood as acceptable within a community. Orm, like other medieval intellectuals, presupposed that speech completed or delivered the written homiletic text and that the purpose of a written text was to trigger and guide oral forms of vernacular scripture through reading aloud to lay and clerical audiences. He retexted existing and French scribal orthography to develop a local orthographic code and embedded his text within an oral ecology rather than separating that text from the speech community. To put it differently, Orm, rather than adopting standardized spelling for particular words, embeds local accent within a text-specific orthography. Orm’s spellings and pronunciations are already multimodal.[2]
            One of the most important insights derived from new critical studies of literacies, medieval and modern, is that mature, sophisticated writing, reading, and thinking are not entirely internal activities, whether we are talking about oral reading and dictation or silent reading and inscription. Literate understandings are products of a circuit from text interface to body to emotions to cognition and back to text. The hinge of reading actually comprises textual fluidity.  In this respect, multimodal acts of writing and reading in multimodal environments (that formulation is not a redundancy) constitute narratives of agentive cognition and affect. At the same time, multimodal environments, with their and presupposing technologies, make up (create) users and meaners within situational parameters.
           Among the traditional senses, medieval intellectuals and religious advisers regarded SIGHT as implicated in and problematized within medieval literacies. Literate bodies and literate minds activate and interrelated sight, hearing, touch, and gesture in various ways. I have discussed elsewhere how medieval literate practices are deeply involved with visual signs and actions, including eye movements over the page, gestures, and touch (Amsler 2001, 2011). In medieval accounts of the senses, sight often combines with or contains touch, and seeing was described as a kind of touching and as the principal frame for understanding (Amsler and Enrique Diaz unpublished). Both oral and silent reading or ruminatio involve body movements, whether those movements are accidental or situational or are eyed to devotional, intellectual, or recreational modes of engaging with texts. The idea of embodied or kinetic reading also leads us to consider reading events in concrete multimodal visual and fields. I’ll address such embodied readings by considering first an intriguing and well known illumination in a mid-fifteenth century English book of hours (BL Harley 2971, fol. 109v) (Fig 1). The Harley 2971 book of hours, produced in Paris circa 1450-1460, contains 165 folios in Latin and French. There are eighteen full page colour miniatures with partial foliate borders. The illustrations serve as textual pragmatic markers of key divisions in a book of hours sequence (Psalms, Office of the Virgin, Office of the Dead, Litany of Saints, etc.; on the composition and format of books of hours, see Duffy 2006 ).
             The image placed on fol. 109v at the beginning of the Office of the Dead in BL Harley 2971 depicts in the foreground a casket draped in black and candles and in the second (medial) visual plane a large liturgical book and the monastic celebrants aligned
horizontally behind the altar. For now, let’s

​IMAGE




(Fig. 1: BL Harley 2971, fol. 109v, 1450-1460)


focus on the figures and the book and how they are related to one another. Given the contents of a book of hours and the illustration’s placement in the text, the image is textually self-referential. It portrays a scene of oral reading and liturgical performance, which the book of hours itself is designed to guide or provide the script for. The image also articulates the materiality of textual production in medieval multimodal literacies. As we will see, for some, the liturgical book is an inscription to be searched through for the language of the requiem. For others, it is a mute sign of Latin religious authority within the performance of the sung liturgy.
           There are seven human religious figures and one large liturgical book in the image. While the image itself is self-referential to the act of reading, the book depicted on a lectern is distinctly different from the smaller hand-held book of hours (210 mm x 150 mm) in which the image is located. The figures around the book are differentiated by clothing, posture, faces, and gestures. On the left, two black robed monks stand solemnly away from the book and are looking not at the book but down to the floor. Four tonsured monks hover around the liturgical book and exhibit different gestures and facial expressions suggesting anxiety or worry. All have closed moths, implying they are not in fact singing at the moment. However, one tonsured monk just to the left of the centre line stands tall, looking away from the book and singing with open mouth and confident facial expression. What’s going on here?
            The single manuscript illustration depicts three different modes and degrees of involvement with the funeral liturgy as praxis: listening silently and solemnly, puzzling through the oral liturgy guided by the written text, and singing the liturgy confidently and apart from the text. The three right-most monks with their anxious facial expressions seem lost in the liturgy text, perhaps unable to find their place or the words. Their hand gestures, postures, and closed mouths imply they need the shared written text in order to participate in the liturgy. The three figures also constitute a visual group. Their heads are slightly lowered and aligned horizontally, the far right figure rests his hand on the shoulder of the figure adjacent to him, and the three are clothed similar. The fourth figure around the book is located on the image’s vertical centre line but triggers a different visual frame. He is wearing eye glasses and is the figure bent closest to the book, perhaps struggling to actually see the letters on the page. Like the other figures around the book, the monk with eye glasses is not singing the liturgy but searching the page.
            The tallest figure in the image is the one explicitly intoning the liturgy. He stands apart from the book with open mouth and confident facial expression, his head pointed upward slightly as he sings. He rests one hand on the shoulder of the figure with eye glasses. This hand gesture reaches across different group spaces and suggests that while the singer is involved orally with the funeral mass, he is also connected with the group hovering around the book.
            So what’s all this got to do with medieval multimodality? In earlier studies of literacy, oral consciousness and practices were sharply distinguished from written consciousness and practices – what became known as the Oral/Literate Divide (see e.g. Havelock, Ong, Goody, and sometimes Levi-Strauss). The so-called “literate mind” was conceived to be qualitatively and cognitively different from the “oral mind.” The “literate mind” is more visually oriented, more able to conceive of and use abstract concepts to think with, and able to understand discourse in decontextualized forms apart from immediate situations of production and reception. In addition, the “literate mind” considers written or printed texts to be linguistically fixed, autonomous objects. People inhabiting primary orality, on the other hand, with little or no experience of written language, display more immediate cognition in concrete situations, are more oriented to sound and movement, and are able to memorize, retain, and reproduce orally extensive amounts of discourse in fluid but variable manners. In the framework of the Oral/Literate Divide, oral cognition and experience is more communal; literate cognition and experience are more individual.
              A better understanding of medieval literacies, contemporary new media, and new semiotically-informed cognitive studies has disrupted, even abolished many of these generalisations about oral and written cognition and consciousness. Studies of both new media and medieval manuscript-based literacy, not to mention Derrida’s grammatology and Barthes’ textual critique, have rendered the idea of a clear, distinct Oral/Literate Divide untenable and cognitively imperialistic. Medieval literacies are often inherently communal. Many manuscript texts were produced through collaborative oral dictation and inscription, were intended for group reading, are multimodally designed with inscription, images, and hypertextual markup, and display a relatively open, if not entirely unfixed textual form which can be altered by retexting, glossing, and elaboration by additional writers and readers over time. While Goody and Ong paid some attention to the particulars of medieval literacies across ten centuries (!), they for the most part regarded western medieval textualities and reading practices as a way station on the road of progress from Primary Orality to Full Literacy.
The BL Harley 2971 book of hours illustration reveals a certain medieval awareness of the complexities and different individual competences within a complex literate mode such as the sung Latin liturgy. Not everyone with access to the controlling text necessarily could access the text linguistically. Some have trouble finding their place or construing the text. Others have trouble literally seeing the text well or at all. People’s physical, cognitive, and affective relations to oral and visual texts change over time as our bodies age or change due to disease, accident, or trauma. The monk wearing eyeglasses in the centre of the manuscript image peers into the book at the centre of the book being read. The visual prosthesis was a new invention of the late Middle Ages and revolutionised literate visual perception for many people, especially those intimately involved with book culture. The first recorded spectacles for an individual to wear and use is from Pisa (Italy) around 1286 (www.antiquespectacles.com/timeline). Literate bodies grow up, mature, and grow old. Our sensory apparatus trained for reading and writing change over time. An often unacknowledged and unproblematised presupposition of much cognition, literacy, and visual art research is that the embodied senses do not alter or decay. So the monk’s eyeglasses function as a signifier of the frailty of the body, whether from genetic deviation or simply the inevitable process of aging. At the same time his eyeglasses also signify how a human prosthesis, a new technology, does not extend (like the telescope) but enable his visual sense to operate competently in the everyday world. His eyeglasses enable to monk to participate in religious literate culture. Judging by his face and the faces of those around him and by the colours and design of his robe matching that of the cloth covering the casket, he is perhaps more engaged and participatory than are some of his fellow monks.
             A ritualized text-based activity such as the Office of the Dead doesn’t necessarily require that all participants pay close visual attention nor be physically proximate to the authoritative text. Given his posture and facial expression, the single singer has memorized well the liturgical text for the funeral mass and become a kind of “living book.” But we can’t know how he intones the words of the liturgy, with what accent, accuracy, or affective voice. There are not indications of the vocal or linguistic sounds he makes, nor are there any indications of how others might be responding to his behavior. The other figures are either in solemn devotion or paying close attention to the liturgical book or puzzled about their interaction with the book. While implying through visual signs linguistic speech and musical sound, the text and image remain mute.
               Embodied literacies cut across religious and secular contexts. Sight, touch, and movement might preempt sound entirely. Le Roman de Flamenca is an anonymous Provencal poem of 8095 lines contained in a single thirteenth-century manuscript (Bibliothèque Municipale de Carcassonne). Le Roman de Flamenca is a hyperliterate narrative which thematizes textual erotics and self-consciousness as well as the pleasures of multimodal literacy. Like Bernard of Clairvaux reading the Song of Songs, the romance foregrounds the erotic kiss, but for romantic rather than spiritual desire. The romance includes several scenes of reading which foreground sexual pleasure and material textuality in multimodal contexts. In one scene, the lover, William of Nevers, uses his breviary as a go-between during the Mass to touch the lips of his beloved, but unfortunately for him married, Flamenca. In a narrative metonymy, William fetishistically yearns for the communal mass book which carries the trace of Flamenca’s touch, her liturgical kiss and fingering, her touch, so that he may look at, hold, and kiss it/her (ed. Blodgett 1995: lines 2587-2610). When the lovers finally kiss one another directly, lips to lips, their erotic joy repeats the affects they experienced when one of them kissed a mass-book the other had touched (lines 6543-51). As in the Paolo and Francesca episode in Dante’s Inferno (Canto 5), the Flamenca narrative reverses the field of representation, so that face-to-face erotic desire re-presents, that is, re-enacts textually mediated desire. The circulating mass book in Flamenca produces secondary presence through absence, not unlike Troilus’ response to Criseyde’s letters in Chaucer’s Trojan romance. Kissing the book is a performance of desire and pleasure through displaced or mediated affectus rather than real presence.
            The romance takes textual erotics, or erotic textuality, a step further when Flamenca’s husband, Sir Archambaut, unknowingly brings his wife a manuscript greeting card from her lover. Alone in her bedroom with her nurse, Flamenca eagerly opens the manuscript. On the page, the narrator says,
There were two figures (ymages) beautifully
figured with such delicacy
they seemed to be fully alive.
And one was in the foreground on his knees
kneeling directly in front of the other.
A flower came out of his mouth
touching the beginnings of the lines (verses);
and at the end there was another
that also gathered and wrapped them all
and guided them together toward the ear
of the other figure where True Love
like an angel gave her advice,
interpreting what the flower meant. (lines 7111-23)
Flamenca’s bedroom scene of reading retexts the Annunciation scene in a book of hours as fin’amors. The episode repeats parodically certain kinds of embodied and eroticized devotional reading. Flamenca immediately recognizes William’s and her own face in the textual figures. Lying in bed, she and her nurse hold the page and fold the edges together like hinges, and the text enacts a kiss:
Often they [Flamenca and her nurse] folded and unfolded them
and were careful not to damage them by rubbing (bregon)
so that neither letters nor pictures (en letras ni em penchura)
look in the least erased (effassadura). (lines 7138-41)
More privately, every night, Flamenca kisses the fetishized text as if it were a relic or a breviary or her lover’s body, but she is careful not to erase or smudge any part. Reading with gestures and touch as well as sight and sound “ages” the material text (words and images) and makes it, that is, the text and not just the image, less adequate or desirable as a visual and material substitute for her absent lover, less of a “whole” image of the absent one. She carefully folds the edges of the manuscript page to produce an ever-perfect textual kiss. The manuscript greeting card and its hidden, performative love note recall Bernard of Clairvaux’s sermon on the Song of Songs, where the earthly “kiss” joins lips to lips while Christ’s “kiss of his mouth” joins the human to the divine.
            In this scene we might forget that the individual manuscript page with its writing, drawing, and mechanics doesn’t feel pleasure or pain or desire. Material pages don’t have emotional lives, but within a multimodal encounter the page is part of an affective experience and cognition. Carrying the visual image of her lover and herself, Flamenca’s manuscript love note becomes the place where desire lands.
            Later, the love note becomes more than a fetish object as Flamenca substitutes the text for her absent lover in her bed:
. . . ‘Friend, I feel your heart
in place of mine where it hides away,
and thus I place these greetings
 so close to it [my heart] so it might feel
and share my pleasure’. (lines 7151-55)
In Flamenca, transgressive affective literacy mimics Christian literate devotion as textual festishization. Flamenca’s embodied literacy also poaches on more general manuscript practices for performative multimodal literacies. Private or silent affective reading provokes Flamenca’s embodied pleasure, from paper to flesh to soul. Literate technology and artistic design bring body and text into a closer affective relationship as the hinges of reading (bodies and pages) are literally folded together. The material manuscript page insists on its being there as part of the literate interaction rather than as a permeable or transparent portal to an imagined world. At the same time, Flamenca maintains a light touch on the page with her lips or fingers, careful not to smudge or erase the image of her lover. Her agency as a desiring reader is constructed around her intense passionate affect toward the manuscript object and her equally intense desire to maintain the eroticized images as eternal, without change. When the Flamenca narrative eroticizes manuscript images and letters, those visual images are revealed to be unstable, impermanent, potentially decaying, always in danger of disappearing or being erased through the devouring sight and touch of desire.
            Despite the increasing textual consciousness of the Middle Ages, medievals tended not to talk much directly about the multimodality of writing or reading. Nonetheless, we encounter numerous scenes of reading and metatextual consciousness: Augustine’s observation of Ambrose reading silently, Alfred learning to read, Old English riddles devoted to books, moths, and bookcases, Anselm of Canterbury on devotion and meditation, the author of the Ancrene Wisse instructing female readers on how their devotional texts can or should be read, Guibert de Nogent reading Ovid under the bed covers at night, Paolo and Francesca reading a Lancelot romance together and getting busy, Margery Kempe worrying if her dictation and writing detract too much from her prayer. Some of these episodes do refer to sensory differences in textual interaction. And of course there are the traditional but problematic assertions in medieval culture that visual images are the “book” of the laity. Augustine’s encounter with Ambrose reading silently comes the closest to reflecting the complexities of multimodal literate experience. Augustine is concerned that Ambrose’s silence before the mute text might suggest illegitimate reading or noncomprehension. He rules out both negative implications but still finds the encounter with Ambrose inscrutable (cf. Amsler 2011). Otherwise, however, we don’t hear much about how different senses interact or how different modes or media interact to create textuality, meaning, or literate performance.
The Roman de Flamenca’s scene of reading is unusual in this regard. The narrative focuses explicitly on the ways in which Flamenca experiences the written and illustrated text, how she handles it with her fingers, lips, and eyes and how it affects and excites her. But the episode (and there are others in the Flamenca text, such as the mass book being passed around during the liturgy) is a fictional representation of the acts of writing and reading. Some extant medieval manuscripts do include cut-outs, movable parts, and so forth, but I haven’t myself located any manuscript that matches up well with the love note described in the romance. But I find it hard to believe that the romance describes a completely imaginary kind of material text. Perhaps others can inform me about what lies in the archives.
            Multimodality, visual, oral, kinetic, was a significant, even fundamental part of medieval literacies, in material texts, reading practices, and composition. Medieval understandings of the senses also informed multimodal text production and reception. Medieval multimodalities challenge us to retext Eurocentrism's universalising discourse of modernity and autonomous knowing, threaded with metaphors of the senses. When we think about multimodality as a concept and as practice in medieval and material contexts, we do well to keep in touch with and keep an eye on and embrace how the human senses and embodiment are structuring structures (not just media) for multimodal meaning making and affective textual interactions.

References
Amsler, Mark. 2010. “Premodern Letters and Textual Consciousness: From the Pre-Socratics to the First Grammatical Treatise.” Historiographia Linguistica, 37:279-319.
--- 2011. Affective Literacies: Writing and multilingualism in the later Middle Ages. Turnhaut: Brepols.
--- and Javier Enrique Díaz-Vera. Unpublished. “In the Realm of the Senses: Language of Sight and Touch in Middle English.” www.auckland.academia.edu/markamsler. Accessed 1 June 2016.
Anselm of Canterbury. 1998.  Monologium. In The Major Works. Ed. Brian Davies and G. R. Evans. Oxford: Oxford University Press.
Augustine. 1962. De doctrina christiana. De vera religion. Ed. K. D. Daur and J. Martin. CCSL, vol. 32. Turnhaut: Brepols.
--- 1954. De magistro. Ed. Klaus-Detlef Daur. CCSL, vol. 29. Turnhaut: Brepols.
--- 1968. De trinitate (Books 13-15)Ed. W. J. Mountain and F. Glorie. CCSL, vol. 50A. Turnhaut: Brepols.
--- 1975. De dialectica. Ed. Jan Pinborg. Trans. Darrell Jackson. Dordrecht: Reidel.
Bacon, Roger. 2013. On Signs (De signis). Trans. Thomas S. Maloney. Toronto: Pontifical Institute of Mediaeval Studies.
Boniface. 1916. Die Briefe des heiligen Bonifatius und Lullus. Ed. M. Tangl. MGH, Epistolae Selectae, vol. 1. Berlin. (1919?)
Castro-Gómez, Santiago. 2005. Le hybris del punto cero: Ciencia, raza e illustración en la Nueva Granada (1750-1816). Bogotá: Editorial Pontificia Universidad Javeriana.
Duffy, Eamon. 2006. Marking the Hours: English People and Their Prayers 1240-1570. New Haven and London: Yale University Press.
First Grammatical Treatise: The Earliest Germanic Phonology. An Edition, Translation, and Commentary. Ed. Einer Haugen. 2nd ed.  London: Longman, 1972.
Goody, Jack. 1986. The Logic of Writing and the Organisation of Society. Cambridge: Cambridge University Press.
Halliday, M. A. K. 1994. An Introduction of Functional Grammar. 2nd ed. London: Edward Arnold.
Halliday, M. A. K. and Matthiessen, C. M. I. M. 2014. Halliday’s Introduction to Functional Grammar. 4th ed. London: Routledge.
Havelock, Eric. 1963. A Preface to Plato. Cambridge, MA: Belknap Press.
Hayles, N. Katherine. 2002. Writing Machines. Cambridge, MA and London. The MIT Press.
Jakobson, Roman. 1960. "Linguistics and Poetics." In Style in Language, ed. Thomas Sebeok. Cambridge, MA: M.I.T. Press. Pp.350-377.
Kilwardby, Robert. 1975. The Commentary on “Priscianus Major” Ascribed to Robert Kilwardby. Ed. Karin M. Fredborg et al. Cahiers de l‘Institut du Moyen-Âge Grec et Latin, Vol. 15:1-143.
Kress, Gunther. 2003. Literacy in the New Media Age. London: Routledge.
Kress, Gunther and van Leeuwen, Theo. 2001. Multimoda Discourse: The modes and media of contemporary communication. London: Edward Arnold.
Le Goff, Jacques. 1988. The Medieval Imagination. Trans. Arthur Goldhammer
Levi-Strauss, Claude. 1966. The Savage Mind. Trans. Unnamed. (originally published as La Pensée sauvage [1962])
Multiliteracies: Literacy Learning and the Design of Social Futures. 2000. Ed. Bill Cope and Mary Kalantzis. London: Routledge. (New London Group)
Nichols, Stephen. 1983. Romanesque Signs: Early Medieval Narrative and Iconography. New Haven: Yale University Press. (Rev. ed. 2011)
Ong, Walter. 1982. Orality and Literacy: Technologising the Word. London and NY: Routledge.
Orm. Ormulum. https://archive.org/stream/ormulumwithnotes01whituoft/ormulumwithnotes01whituoft_djvu.txt (digital version of Holt and White’s 1878 edition). Accessed 1 June 2016.
Poyatis, F. 2002. Nonverbal Communication Across Disciplines, Vol. 1: Culture, Sensory Interaction, Speech, Conversation. Amsterdam: John Benjamins.
The Romance of Flamenca (Le Roman de Flamenca)1995. Ed and trans. E. D. Blodgett. NY and London: Garland.
Rosier, Irène. 1994. La parole comme acte: Sur la grammaire et la sémantique au xiiiesiècle. Paris: Librairie Philosophique J. Vrin.
Saenger, Paul. 1997. Space Between Words: Origins of Silent Reading. Palo Alto: Stanford University Press.
Scollon, R. and Scollon, S. 2003. Discourses in Place: Language in the material world. NY: Routledge.
Scribner, Sylvia and Cole, Michael. 1981. The Psychology of Literacy. Cambridge, MA: Harvard University Press.
Vance, Eugene. 1989. Mervelous Signals: Poetics and Sign Theory in the Middle Ages. Lincoln: University of Nebraska Press.
Wertsch, James. 1993. Voices of the Mind: Sociocultural approach to mediated action. Cambridge, MA: Harvard University Press.
www.antiquespectacles.com/timeline/timeline.htm. Accessed 1 June 2016.
Zumthor, Paul. 1973. Essai de poétique médiévale. Paris: Seuil.
 
 
[1] For example, Orm initially used eo and e inconsistently for words such as beon and kneow, which had been spelled with eo in Old English. However, at line 13000 he seems to have changed his mind and then revised his text and changed all eo spellings to e alone (ben and knew), presumably to reflect the current local monophthongized pronunciation.
[2] The discrimination implicit in some forms of spelling and orthography is endemic within literacy programmes in and outside of school. Such linguistic discrimination also depends on particular uses of multimodal textuality. British and US English standardized spelling often conceals, elides, or normatively erases regional, nonnormative, or nonmainstream pronunciation. Southern US children learning to read are asked to orally distinguish PIN from PEN according to their pronunciation for the biro object illustrated in their workbooks. For many of these children, PIN and PEN are homophones: [pIn]. The mainstream US pronunciation of PEN is [pεn].

This page has paths:

This page references: