Novel Speech-Based Emotion Climate Recognition in Peers’ Conversations Incorporating Affect Dynamics and Temporal Convolutional Neural Networks
Peers’ conversation provides a domain of rich emotional information. The latter, apart from facial and gestural expressions, it is also naturally conveyed via peers’ speech, contributing to the establishment of a dynamic emotion climate (EC) during their conversational interact...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10839385/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832576749199687680 |
|---|---|
| author | Ghada Alhussein Mohanad Alkhodari Ahsan H. Khandoker Leontios J. Hadjileontiadis |
| author_facet | Ghada Alhussein Mohanad Alkhodari Ahsan H. Khandoker Leontios J. Hadjileontiadis |
| author_sort | Ghada Alhussein |
| collection | DOAJ |
| description | Peers’ conversation provides a domain of rich emotional information. The latter, apart from facial and gestural expressions, it is also naturally conveyed via peers’ speech, contributing to the establishment of a dynamic emotion climate (EC) during their conversational interaction. Recognition of EC could provide an additional source in understating peers’ social interaction and behavior on top of peers’ actual conversational content.We propose a novel approach for speech-based EC recognition, termed AffECt, which combines peers’ complex affect dynamics (AD) with deep features extracted from speech signals using Temporal Convolutional Neural Networks (TCNNs). AffECt was tested and cross-validated on data drawn from there open datasets, i.e., K-EmoCon, IEMOCAP, and SEWA, in terms of EC arousal/valence level classification. The experimental results have shown that AffECt achieves EC classification accuracy up to 83.3% and 80.2% for arousal and valence, respectively, clearly surpassing the results reported in the literature, exhibiting robust performance across different languages. Moreover, there is a distinct improvement when the AD are combined with the TCNN, compared to the baseline deep learning approaches. These results demonstrate the effectiveness of AffECt in speech-based EC recognition, paving the way for many applications, e.g., in patients’ group therapy, negotiations, and emotion-aware mobile applications. |
| format | Article |
| id | doaj-art-3bcd72fb793b456d8cf5ab2a3556061f |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-3bcd72fb793b456d8cf5ab2a3556061f2025-01-31T00:00:48ZengIEEEIEEE Access2169-35362025-01-0113167521676910.1109/ACCESS.2025.352912510839385Novel Speech-Based Emotion Climate Recognition in Peers’ Conversations Incorporating Affect Dynamics and Temporal Convolutional Neural NetworksGhada Alhussein0https://orcid.org/0000-0001-6181-8306Mohanad Alkhodari1https://orcid.org/0000-0002-5248-6327Ahsan H. Khandoker2https://orcid.org/0000-0002-0636-1646Leontios J. Hadjileontiadis3https://orcid.org/0000-0002-9932-9302Department of Biomedical Engineering and Biotechnology, Khalifa University of Science and Technology, Abu Dhabi, United Arab EmiratesRadcliffe Department of Medicine, Cardiovascular Clinical Research Facility, University of Oxford, Oxford, U.K.Department of Biomedical Engineering and Biotechnology, Khalifa University of Science and Technology, Abu Dhabi, United Arab EmiratesDepartment of Biomedical Engineering and Biotechnology, Khalifa University of Science and Technology, Abu Dhabi, United Arab EmiratesPeers’ conversation provides a domain of rich emotional information. The latter, apart from facial and gestural expressions, it is also naturally conveyed via peers’ speech, contributing to the establishment of a dynamic emotion climate (EC) during their conversational interaction. Recognition of EC could provide an additional source in understating peers’ social interaction and behavior on top of peers’ actual conversational content.We propose a novel approach for speech-based EC recognition, termed AffECt, which combines peers’ complex affect dynamics (AD) with deep features extracted from speech signals using Temporal Convolutional Neural Networks (TCNNs). AffECt was tested and cross-validated on data drawn from there open datasets, i.e., K-EmoCon, IEMOCAP, and SEWA, in terms of EC arousal/valence level classification. The experimental results have shown that AffECt achieves EC classification accuracy up to 83.3% and 80.2% for arousal and valence, respectively, clearly surpassing the results reported in the literature, exhibiting robust performance across different languages. Moreover, there is a distinct improvement when the AD are combined with the TCNN, compared to the baseline deep learning approaches. These results demonstrate the effectiveness of AffECt in speech-based EC recognition, paving the way for many applications, e.g., in patients’ group therapy, negotiations, and emotion-aware mobile applications.https://ieeexplore.ieee.org/document/10839385/Conversational emotion climate recognitionarousal/valencemel-frequency cepstrum coefficients (MFCCs)affect dynamicstemporal convolutional neural networks (TCNN)AffECt |
| spellingShingle | Ghada Alhussein Mohanad Alkhodari Ahsan H. Khandoker Leontios J. Hadjileontiadis Novel Speech-Based Emotion Climate Recognition in Peers’ Conversations Incorporating Affect Dynamics and Temporal Convolutional Neural Networks IEEE Access Conversational emotion climate recognition arousal/valence mel-frequency cepstrum coefficients (MFCCs) affect dynamics temporal convolutional neural networks (TCNN) AffECt |
| title | Novel Speech-Based Emotion Climate Recognition in Peers’ Conversations Incorporating Affect Dynamics and Temporal Convolutional Neural Networks |
| title_full | Novel Speech-Based Emotion Climate Recognition in Peers’ Conversations Incorporating Affect Dynamics and Temporal Convolutional Neural Networks |
| title_fullStr | Novel Speech-Based Emotion Climate Recognition in Peers’ Conversations Incorporating Affect Dynamics and Temporal Convolutional Neural Networks |
| title_full_unstemmed | Novel Speech-Based Emotion Climate Recognition in Peers’ Conversations Incorporating Affect Dynamics and Temporal Convolutional Neural Networks |
| title_short | Novel Speech-Based Emotion Climate Recognition in Peers’ Conversations Incorporating Affect Dynamics and Temporal Convolutional Neural Networks |
| title_sort | novel speech based emotion climate recognition in peers x2019 conversations incorporating affect dynamics and temporal convolutional neural networks |
| topic | Conversational emotion climate recognition arousal/valence mel-frequency cepstrum coefficients (MFCCs) affect dynamics temporal convolutional neural networks (TCNN) AffECt |
| url | https://ieeexplore.ieee.org/document/10839385/ |
| work_keys_str_mv | AT ghadaalhussein novelspeechbasedemotionclimaterecognitioninpeersx2019conversationsincorporatingaffectdynamicsandtemporalconvolutionalneuralnetworks AT mohanadalkhodari novelspeechbasedemotionclimaterecognitioninpeersx2019conversationsincorporatingaffectdynamicsandtemporalconvolutionalneuralnetworks AT ahsanhkhandoker novelspeechbasedemotionclimaterecognitioninpeersx2019conversationsincorporatingaffectdynamicsandtemporalconvolutionalneuralnetworks AT leontiosjhadjileontiadis novelspeechbasedemotionclimaterecognitioninpeersx2019conversationsincorporatingaffectdynamicsandtemporalconvolutionalneuralnetworks |