Multimodality research is an emerging discipline that studies the way human communication naturally relies on appropriate combinations of multiple 'modes' of expression. Recognising the fundamental role of multimodality has caused fields concerned with human communication and interaction to extend their scope across traditional disciplinary borders. Given the rapidly growing interest in multimodality across diverse fields of study, there is now an urgent need for a robust theoretical foundation for multimodality research. However, the vast majority of multimodality research remains based on limited data and hand-picked examples, despite long-standing calls for establishing a closer coupling between theory and data.
This project lays a foundation for conducting data-driven empirical research on multimodality in the domain of everyday cultural artefacts, such as magazines, social media videos, textbooks and news broadcasts. The goal of the project is to provide a robust theoretical foundation and a reproducible methodology for supporting empirical research on the multimodality of everyday cultural artefacts. Whereas the theoretical advances aim to provide a robust, empirically-backed scaffolding for analysing multimodality in all domains in which the phenomenon is of concern, the reproducible methodology developed in the project enables researchers to build increasingly large corpora, which maintain a level of analytical depth needed for analysing how humans make and exchange meanings multimodally.
Develop new methods for creating large, reliable and reproducible multimodal corpora by using non-expert workers on online crowdsourcing platforms for collecting human insights at scale
Develop novel methods for multimodal corpus analysis by leveraging neuro-symbolic artificial intelligence, which allows combining human insights on multimodality with the pattern recognition capability of artificial neural networks
Use the novel corpora and methods to interrogate key concepts in multimodality theory and to model their contributions to multimodal discourse, thus renewing core theories of multimodal meaning-making
For an up-to-date list of publications and other research outputs, see the University of Helsinki research portal.