Multimodality research is an emerging discipline that studies the way human communication naturally relies on appropriate combinations of multiple 'modes' of expression. Recognising the fundamental role of multimodality has caused fields concerned with human communication and interaction to extend their scope across traditional disciplinary borders. Given the rapidly growing interest in multimodality across diverse fields of study, there is now an urgent need for a robust theoretical foundation for multimodality research. However, the vast majority of multimodality research remains based on limited data and hand-picked examples, despite long-standing calls for establishing a closer coupling between theory and data.
This project lays a foundation for conducting data-driven empirical research on multimodality in the domain of everyday cultural artefacts, such as magazines, social media videos, textbooks and news broadcasts. The goal of the project is to provide a robust theoretical foundation and a reproducible methodology for supporting empirical research on the multimodality of everyday cultural artefacts. Whereas the theoretical advances aim to provide a robust, empirically-backed scaffolding for analysing multimodality in all domains in which the phenomenon is of concern, the reproducible methodology developed in the project enables researchers to build increasingly large corpora, which maintain a level of analytical depth needed for analysing how humans make and exchange meanings multimodally.
For an up-to-date list of publications and other research outputs, see the University of Helsinki research portal.