M.Sc. Imtiaj Ahmed defends his doctoral thesis "Haptic interaction with affective agents in virtual reality" on Thursday the 1st of February 2024 at 13 o'clock in the University of Helsinki Exactum building, Auditorium B123 (Pietari Kalmin katu 5, 1st floor). His opponent is Assistant Professor Gijs Huisman (Delft University of Technology, Netherlands) and custos Professor Giulio Jacucci (University of Helsinki). The defence will be held in English.
The thesis of Imtiaj Ahmed is a part of research done in the Department of Computer Science and in the Ubiquitous Interaction group at the University of Helsinki. His supervisors have been Professor Giulio Jacucci (University of Helsinki) and Associate Professor Michiel M. Spapé (University of Macau, Macau SAR).
Haptic interaction with affective agents in virtual reality
As the future demands more user-agent social interaction, artificial agents are increasingly appearing in virtual reality (VR). These agents can be designed to utilize emotional facial expressions and touch in order to enhance the affective experience in virtual reality (VR). Previously, researchers have investigated the effectiveness of touch and facial expressions in influencing user emotions and behaviors and the multisensory integration of tactile and visual cues. This has been done using a limited array of haptics and presenting the tactile and visual cues originating from separate unembodied sources. However, in real life as well as immersive VR people expect touch and emotional expression from the same body during interpersonal communication. Our understanding of the effectiveness of different haptic technologies in supporting affective interaction with embodied virtual agents in VR remains limited as does our understanding of the crossmodal integration between visual and tactile social cues.
This investigation is based on five studies to explore the process and impact of haptics for affective interaction with virtual agents in VR. The VR scenarios employed in the studies are designed to elicit affective responses utilizing expressive virtual agents. These agents can show emotional facial expressions and touch the user or let the user touch them. The objectives are to identify haptic feedback technologies that effectively influence user emotions and touch experiences in affective VR, to deal with the physiological signals integration challenges, and to gain a detailed understanding of multimodal touch perception and expression.
The investigation advances the understanding of the optimal haptics for affective communication and cross-modal integration between visual and tactile stimuli during user-agent social interaction in VR. The findings suggest that mechanical pressure-based haptic actuators are perceived as more pleasant, natural, and effective in promoting emotional interdependence and co-presence for affective interaction in VR compared to vibrotactile actuators. The emotional expressions of virtual agents influenced users' emotions, as well as how they perceived and expressed touch. Additionally, users' emotional states can be predicted from their haptic responses. Regarding physiological sensing, the results revealed that electroencephalogram (EEG) can be robustly used with a head-mounted device without fear of electrical interference. Moreover, simpler electrophysiological signals like facial Electromyography (fEMG), electrodermal activity (EDA), and accelerometer can reliably detect changes in behavioral and emotional states in exergaming affective VR. The dissertation also addresses other integration challenges, such as marker signal transmission, and synchronization of markers and stimuli. Furthermore, it provides a detailed process description for building effective haptic interfaces and affective agents, along with validation results. The outcome of this research can be valuable for researchers and practitioners interested in enhancing the emotional adaptability of HCI systems with haptic technology.
Availability of the dissertation
An electronic version of the doctoral dissertation will be available on the e-thesis site of the University of Helsinki at http://urn.fi/URN:ISBN:978-952-84-0079-0.
Printed copies will be available on request from Imtiaj Ahmed: imtiaj.ahmed@helsinki.fi