MULTIMODAL PSYCHOLOGICAL STRESS DETECTION USING ATTENTION-BASED FEATURE ALIGNMENT AND DEEP LEARNING

Abstract

This paper proposes a novel framework for detecting psychological stress through multimodal data fusion. In this study we developed a novel framework that accommodates behavioural measurements (e.g. keyboards tap), biometric measures (e.g. electrodermal activity), and self-reported measures (e.g. Likert response to stress) to provide an accurate measure of reliable stress in various contexts. We leverage a hierarchical neural attention framework that learns the temporal dependencies and relationships between modalities (i.e. all of the above). Our tests on publicly available datasets showed an accuracy of 89.7% when classifying people as being stressed using the elaborations from the method we provided in this study, which represents a distinct improvement on unimodal measurements, and prior multimodal approaches. The ability of our framework to generalize across a wide variety of modes of evaluation has tremendous implications for real world applications, especially in mental health monitoring and intervention. To promote future research in this area we have provided our solution as an open-source library.