27 January 2011

Multimodal Situation Models

The individual instantiations of cultural models are carried out by personal and situation models (Johnson-Laird, 1983; van Dijk & Kintsch, 1983; Zwaan & Radavansky, 1998). Situation models (van Dijk & Kintsch, 1983; Zwaan & Radavansky, 1998) have a marked semantic nature because enable us to construct and represent meanings from what is expressed, conveyed and perceived by sounds (and music), writing, visual images, eye-gaze, hand-pointing, touch, face expressions and other body movements as part of everyday action and experience. To avoid terminological confusion, ‘semantic’ is used here to refer to capacity of situation models to construct and represent meaning, not to ‘semantic memory’. That is, the term ‘semantic’ here applies to the meaning, reference, and discourse semantics, not to the socially-shared knowledge organized in semantic memory.

We construct and update multimodal situation models when we participate in and interpret activities (e.g. talking about past events, watching the news on TV). Situation models reconstruct and represent specific situations according to personal interpretations of such events. They are organized by a schematic structure consisting of a number of fixed and schematic categories, e.g. temporal and spatial setting (in a various levels of specificity according to their concrete relevance), on-going activities (talking about past events), participants, roles, identities and relationship (including the self), opinions and emotions, knowledge and intentions and goals (van Dijk, 1999). Situation models are reconstructed and represented in episodic memory and supported by socially-shared knowledge and emotional codes driven by cultural models.

The construction and updating of these mental models is determined by the cognitive processes of event segmentation, understanding of utterance, reconstruction of old models of similar situations in episodic memory (van Dijk & Kintsch, 1983; Zwaan & Madden, 2004), and instantiations of more general personal information, as well as instantiations of multimodal cultural models. Situation models create the grounds for everyday action (e.g. buy a coffee to take away) and embodied interaction and communication (e.g. family members sharing memories at the dinner table) and operate to top-down segment activities as discrete units (van Dijk & Kintsch, 1983). It is important to point out that we do not construct situation models from scratch. The reconstruction and updating of models (e.g. my first full English breakfast in London) implies the re-activation and re-shaping of old models in episodic memory (e.g. my usual breakfast in my hometown Buenos Aires, basically consisting of a strong café latte and two small croissants).

The reconstruction and updating of situation models in episodic memory is largely driven by working memory processes (Baddeley, 2007; Cowan, 2005). These online processes play a central role in intertwining our brains, bodies and the material and social environment, and thereby, creating the conditions for ‘being-in-the-world’ (Heidegger, 1962). Working memory processes enable us to perceive, decode and interpret these environments, represent the temporally stored information resulting of such processes, as well as retrieve, reconstruct and update situations models in episodic memory (van Dijk, forthcoming). Thus, this new information is kept accessible for further processing (Ericsson & Kintsch, 1995; Zwaan & Radvansky, 1998) and enables the efficient integration of information from different modalities in situation models.

Several studies in text comprehension (Zwaan & Madden, 2004) showed that readers keep an integrated situation model in episodic memory while the current model is being constructed by working memory processes. In this way, as Zwaan and Radvansky claimed ‘the updating occurs by forming links between the current model and the retrieved elements of the integrated model’ (1998: 166). This online process keeps running in parallel until we reconstruct and update a complete model in episodic memory. However, we must keep in mind that new situation models are formed only when there are changes in relation to the schematic categories (e.g. setting, participants, intentions and goals, etc.), which affect processing and interaction. These changes are motivated by the processes of event segmentation.


Baddeley, AD. (2007). Working Memory, Thought and Action. Oxford. Oxford University Press.

Cowan, N. (2005). Working Memory Capacity. Hove, UK: Psychology Press.

Ericsson, K.A. & Kintsch, W. (1995). Long-term working memory. Psychological Review 102, 211-245.

Heidegger, M. (1962). Being and Time (J. Macquarrie & E. Robinson trans.). London: SCM Press.

Johnson-Laird, P. N. (1983). Mental Models. Towards a Cognitive Science of Language, Inference, and Consciousness. Cambridge, MA: Harvard University Press.

van Dijk, T.A. (1999). Context and experience models in discourse processing. In H. van Oostendorp & S. Goldman (eds.), The Construction of Mental Representation during Reading (pp. 123-148). Hillsdale, NJ : Erlbaum.

van Dijk, T. A. (forthcoming). Discourse and Knowledge.

van Dijk, T.A. & Kintsch, W. (1983). Strategies of Discourse Comprehension. New York: Academic Press.

Zwaan, R.A. & Radvansky, G.A. (1998). Situation models in language and memory. Psychological Bulletin 123 162-185.

Zwaan, R. A. & Madden, C.J. (2004). Updating situation models. Journal of Experimental Psychology : Learning, Memory and Cognition, 30, 283-288.

No comments: