We study how human communication naturally combines multiple modes of expression. Whereas face-to-face interaction involves coordinated combinations of spoken language, gestures, posture and gaze, newspapers and textbooks combine written language with diagrams, photographs, illustrations and layout, to name just a few examples.
This phenomenon, multimodality, is an inherent feature of human communication and interaction, but our understanding of how appropriate combinations of modes are constructed remains limited.
We approach this phenomenon from the perspective of multimodal semiotics. This means that we focus on how different modes of expression are used for making and exchanging meanings. To study how meanings are construed multimodally, we combine theories of meaning-making developed in the field of semiotics with linguistically-inspired methods for supporting data-driven empirical research.
To conduct empirical and data-driven research on multimodality, we create corpora, which refers to collections of multimodal data that have been annotated for their features. We develop new ways of describing multimodal phenomena in a systematic manner and explore how novel methods such as crowdsourcing and machine learning can be used to scale up the size of multimodal corpora.
Current multimodal corpora, which remain small due to the time-consuming nature of manual annotation, are insufficient for identifying patterns that define how modes of expression are used in different communicative situations. To this end, we develop novel methods for searching for patterns that define particular forms of multimodal communication, which combine human insights with the pattern recognition capability of machine learning algorithms.
We also conduct research that may be placed under the umbrella of digital humanities. In addition to exploring how theories of multimodality may inform research in the field of digital humanities and beyond, we critically examine novel methods such as crowdsourcing and their role as a part of the academic research infrastructure.
We also collaborate with the Digital Geography Lab in the Department of Geosciences and Geography on studying human–nature interactions and urban multilingualism.