Emotion Recognition With Audio, Video, EEG, and EMG: A Dataset and Baseline Approaches

Authors

Jin Chen, CUNY City College
Tony Ro, CUNY Graduate CenterFollow
Zhigang Zhu, CUNY City CollegeFollow

Document Type

Article

Publication Date

1-26-2022

Abstract

This paper describes a new posed multimodal emotional dataset and compares human emotion classification based on four different modalities - audio, video, electromyography (EMG), and electroencephalography (EEG). The results are reported with several baseline approaches using various feature extraction techniques and machine-learning algorithms. First, we collected a dataset from 11 human subjects expressing six basic emotions and one neutral emotion. We then extracted features from each modality using principal component analysis, autoencoder, convolution network, and mel-frequency cepstral coefficient (MFCC), some unique to individual modalities. A number of baseline models have been applied to compare the classification performance in emotion recognition, including k-nearest neighbors (KNN), support vector machines (SVM), random forest, multilayer perceptron (MLP), long short-term memory (LSTM) model, and convolutional neural network (CNN). Our results show that bootstrapping the biosensor signals (i.e., EMG and EEG) can greatly increase emotion classification performance by reducing noise. In contrast, the best classification results were obtained by a traditional KNN, whereas audio and image sequences of human emotions could be better classified using LSTM.

Comments

This article was originally published in IEEE Access, available at https://doi.org/10.1109/ACCESS.2022.3146729

This work is distributed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).