Abstracts

Download list of abstracts (PDF, 827KB)

Plenary abstracts

Grenzgänger: When modalities merge

John Bateman, University of Bremen

http://www-user.uni-bremen.de/~bateman (opens an external site)

One of the most promising ways of exploring how modalities make the meanings they do is by exploring what happens when they combine or come to influence one another. Such considerations lead naturally to a progressive decomposition of the notion of semiotic mode and to a more refined view of just what is combined in instances of `multimodality'. Many traditional `modalities' show themselves to consist of diverse facets and, in interaction with other modes, typically not all of these facets are engaged at once. Examining cases where modes combine therefore provides a window on what is taken up and why: modes appear to need to be, on the one hand, sufficiently different that there is a semiotic advantage gained in the composition but also, on the other, sufficiently similar to allow the composition to take place. Finding ways of characterising these dimensions of difference and similarity offers a fruitful research direction for refining our understanding of multimodality in general. In this talk I focus on some well known examples of this phenomenon, beginning with the moves out of photography and theatre into film and continuing with the current web of interactions across film and graphic novels. This will suggest that we can usefully go beyond some old boundaries, such as that between static and dynamic artefacts or between hypermedia and `linear' presentations, building some foundations for a more inclusive view of modality on the way.

What the Eye Can Hear

Ross Gibson, Professor of Contemporary Arts, Sydney College of the Arts, University of Sydney

www.rossgibson.com.au (opens an external site)

This paper draws on studies of encounters with remote Indigenous Australian cultures by scholars such as Barry Hill and Martin Thomas. I also refer to my own studies of the transactions between Indigenous and invasive cultures around Sydney, 1788 - 1791. Again and again in Europeans' first-contact accounts of Indigenous cultures, there is a sense that Indigenous knowledge is arranged in people and in space and time in ways that are completely at odds with Western presumptions about cognition and interpretation. There is a sense that awareness is a multi-modal experience and that there is no point in separating the senses that Western commonsense tends to distinguish and hierarchise. While I do not purport to explain Indigenous thought here, I do want to dwell on the enigmas that arise in accounts of the encounters between the indigenous and incursive mentalities. What fundamental questions do the records of these encounters pose for Western assumptions about the operations of sense and the separated senses?

Digital multimodal experiences of the museum visitor

Carey Jewitt, University of London

http://www.ioe.ac.uk/staff/LKLB_25.html (opens an external site)

The last decade has witnessed a rapid and continuous change in museum relationships with visitors. New technology or ‘new’ media have been a key aspect of the reconfiguration of this relationship with museums, visitor experiences, and relationships to knowledge. Museums have used new digital technologies in attempts to improve audience engagement, dialogic exchange and to support interpretation of artefacts and provide access for broader audiences. In this presentation I will explore some of the concerns about issues of authenticity, authority, and ownership that are raised by the use of digital technologies in museums and galleries as well as how issues of interactivity, participation and engagement are realized in the museum through digital technology.

I will focus on four case studies on how the use of digital technology in museums or galleries impacts on the visitor experience selected to describe a range of technologies in use.

Case study 1. You Tube and Flickr: The Weather Project by Olafur Ellliason at the Tate Modern Museum

Case study 2. Interactive artefacts, virtual tours and Websites: The Winston Churchill Museum

Case study 3. Digital visitor trails: OOLK at the D-Day museum

Case study 4: Interactive website: The Leonardo da Vinci experience at the Victoria and Albert Museum

Through these case studies with a focus on specific exhibits this presentation will explore the three questions and associated themes outlined below:

1. How can the use of digital technology in museum or gallery create interactive experiences for the visitor? Interactivity as a concept is explored across the case studies to investigate the range of interactive experiences designed for (and by) the visitor. The case studies describe the character and form of such interactive experiences.
2. In what ways does technology afford new routes to Engagement and participation with an exhibit or museum/gallery? The case studies highlight some of the opportunities in the ways that technology can be used to create and support forms of dialogic exchange and participation.
3. How can digital technology impact on opportunities for visitor interpretation or understanding of exhibits and experiences? The case studies draw attention to some of the ways that the use of digital technology can shape the processes of interpretation and understanding. In particular the ways in which technology can affect the authoritative character of the official knowledge of the museum or gallery. The value and status of visitor knowledge, the potential authority of visitor interpretations, in the contemporary digital spaces is also explored.

Semantic relations between body motion and speech: a commentary with examples

Adam Kendon, University of Pennsylvania

http://www.semioticon.com/semiotix/semiotix9/sem-9-03.html (opens an external site)

How are speakers' visible body movements deemed to have semantic import? How are such movements organized in relation to spoken expressions within utterances? Examples and their implications for how 'gesture', 'speech' and 'language' may be related are presented and discussed.

Life as a Theme: complementarities of verbiage and image in academic discourse

J R Martin, Department of Linguistics, University of Sydney

Presentation slides (PDF, 4.4MB)

In a world of rapidly evolving electronic communication technology, multimodal scholars have been busy documenting and theorising a purported shift from mainly verbal to mainly non-verbal transmissions. Wary though I am of the auto-ethnographic turn in the narrative turn in the discursive turn in the linguistic turn in the humanities and social sciences, in this paper I'll examine the ontogenesis of my own powerpoint presentations as I shift in 2005 from OHP projections and A4 handouts to e-slide projections.

Over time, the role of printed verbiage appears to shift from a mainly ideational knowledge construing function to a mainly textual image and example scaffolding one - complemented of course by spoken verbiage co-instantiating the slides (which I will not consider in this presentation). Of special concern will be the increasing use of diagrams to synthesise vertical discourse in synoptic visualisations, alongside the facility with which multimodal data can be deployed for illustrative and interpersonal effects.

References

_{Bernstein, B. (1999). Vertical and horizontal discourse: An essay. British Journal of the Sociology of Education, 20(2), 157-173.}

_{Martin, J R 1993 Life as a noun. M A K Halliday & J R Martin. Writing Science: literacy as discursive power. London: Falmer (Critical Perspectives on Literacy and Education) 1993. 221-267.}

_{Hood, S 2008 Summary writing in academic contexts: implicating meaning in processes of change. Linguistics and Education 19. 351-365.}

_{Martin, J R 2010 Semantic variation: modelling system, text and affiliation in social semiosis. Bednarek, M & J R Martin [Eds.[ 2010 New Discourse on Language: functional perspectives on multimodality, identity, and affiliation. London: Continuum.1-34.}

_{Zappavigna, M, P Dwyer & J R Martin 2010 Visualising Appraisal Prosody. A Mahboob & N Knight [Ed.] Appliable Linguistics: texts, contexts, and meanings. London: Continuum. 2010. 150-167.}

Image/text relations in multimodal comprehension and composition: Challenges for an emerging Australian National Curriculum, classroom teaching and social semiotic research

Len Unsworth, University of New England

http://www.une.edu.au/staff/lunswort.php (opens an external site)

Of the many challenges concerning multimodal literacy, this session will focus on the nature of image/language interaction in texts, or, to put it another way, the construction of meaning at the intersection of image and language. Firstly, we will note the rhetoric of the importance of multimodal literacy in former State and now National Curriculum documents. Secondly, we will examine the reality of multimodal reading comprehension assessment, as it occurred in the former NSW Basic Skills Tests (BST) and now in the Australian National Assessment Program in Literacy and Numeracy (NAPLAN). This section will summarize results of a recent ARC Linkage Project drawing attention to the relative difficulty children experience in comprehending different kinds of image-language interaction in literacy tests and the paucity of attention to image-language interaction in the NAPLAN. Thirdly, we will reflect on the hegemonic mono-modal approach to text composition in schooling, at least in powerful institutional contexts of writing assessment, in contrast to the increasingly common strategic deployment of image-language interaction in children’s multimodal composition with paper and digital media – and particularly the role of different types of image-language interaction in children’s compositional strategies. Finally, it seems important to note that, notwithstanding the considerable increase in research in this area in recent years, the theorizing of image-language interaction remains in its infancy. In acknowledging the significant contribution of those whose work derives from Systemic Functional Linguistics, it is clear that much remains to be done, especially from the perspective of an educational semiotics, which would aim to produce accounts of image-language interaction that are accessible and productive in pedagogic contexts.

Listening

Theo van Leeuwen, Faculty of Arts and Social Sciences, University of Technology, Sydney

In this paper I will explore listening as an active semiotic practice. I will characterize it as a form of accompaniment for verbal or musical monologues or solos that combines rhythmic synchronization with and involvement in the monologue or solo with an active interpretation that can be understood by analysing the placement of listening signals, and an evaluation that can be understood by analysing the characteristics of the listening signals. Examples will include the use of verbal and non-verbal listening signals in different kinds of conversation, the use of listening shots in film and television dialogue, and the irregular placement of rhythmic accents in the percussive (piano and drums) accompaniment of jazz solos. The latter part of the paper will involve an element of live musical performance.