Improved BCI calibration in multimodal emotion recognition using heterogeneous adversarial transfer learning
The use of brain-computer interface (BCI) technology to identify emotional states has gained significant interest, especially with the rise of virtual reality (VR) applications. However, the extensive calibration required for precise emotion recognition models presents a significant challenge, parti...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
PeerJ Inc.
2025-01-01
|
| Series: | PeerJ Computer Science |
| Subjects: | |
| Online Access: | https://peerj.com/articles/cs-2649.pdf |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832591207663927296 |
|---|---|
| author | Mehmet Ali Sarikaya Gökhan Ince |
| author_facet | Mehmet Ali Sarikaya Gökhan Ince |
| author_sort | Mehmet Ali Sarikaya |
| collection | DOAJ |
| description | The use of brain-computer interface (BCI) technology to identify emotional states has gained significant interest, especially with the rise of virtual reality (VR) applications. However, the extensive calibration required for precise emotion recognition models presents a significant challenge, particularly for sensitive groups such as children, elderly, and patients. This study presents a novel approach that utilizes heterogeneous adversarial transfer learning (HATL) to synthesize electroencephalography (EEG) data from various other signal modalities, reducing the need for lengthy calibration phases. We benchmark the efficacy of three generative adversarial network (GAN) architectures, such as conditional GAN (CGAN), conditional Wasserstein GAN (CWGAN), and CWGAN with gradient penalty (CWGAN-GP) within this framework. The proposed framework is rigorously tested on two conventional open sourced datasets, SEED-V and DEAP. Additionally, the framework was applied to an immersive three-dimensional (3D) dataset named GraffitiVR, which we collected to capture the emotional and behavioral reactions of individuals experiencing urban graffiti in a VR environment. This expanded application provides insights into emotion recognition frameworks in VR settings, providing a wider range of contexts for assessing our methodology. When the accuracy of emotion recognition classifiers trained with CWGAN-GP-generated EEG data combined with non-EEG sensory data was compared against those trained using a combination of real EEG and non-EEG sensory data, the accuracy ratios were 93% on the SEED-V dataset, 99% on the DEAP dataset, and 97% on the GraffitiVR dataset. Moreover, in the GraffitiVR dataset, using CWGAN-GP-generated EEG data with non-EEG sensory data for emotion recognition models resulted in up to a 30% reduction in calibration time compared to classifiers trained on real EEG data with non-EEG sensory data. These results underscore the robustness and versatility of the proposed approach, significantly enhancing emotion recognition processes across a variety of environmental settings. |
| format | Article |
| id | doaj-art-0497a5fe64894489a7da77b5a341789e |
| institution | Kabale University |
| issn | 2376-5992 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | PeerJ Inc. |
| record_format | Article |
| series | PeerJ Computer Science |
| spelling | doaj-art-0497a5fe64894489a7da77b5a341789e2025-01-22T15:05:20ZengPeerJ Inc.PeerJ Computer Science2376-59922025-01-0111e264910.7717/peerj-cs.2649Improved BCI calibration in multimodal emotion recognition using heterogeneous adversarial transfer learningMehmet Ali SarikayaGökhan InceThe use of brain-computer interface (BCI) technology to identify emotional states has gained significant interest, especially with the rise of virtual reality (VR) applications. However, the extensive calibration required for precise emotion recognition models presents a significant challenge, particularly for sensitive groups such as children, elderly, and patients. This study presents a novel approach that utilizes heterogeneous adversarial transfer learning (HATL) to synthesize electroencephalography (EEG) data from various other signal modalities, reducing the need for lengthy calibration phases. We benchmark the efficacy of three generative adversarial network (GAN) architectures, such as conditional GAN (CGAN), conditional Wasserstein GAN (CWGAN), and CWGAN with gradient penalty (CWGAN-GP) within this framework. The proposed framework is rigorously tested on two conventional open sourced datasets, SEED-V and DEAP. Additionally, the framework was applied to an immersive three-dimensional (3D) dataset named GraffitiVR, which we collected to capture the emotional and behavioral reactions of individuals experiencing urban graffiti in a VR environment. This expanded application provides insights into emotion recognition frameworks in VR settings, providing a wider range of contexts for assessing our methodology. When the accuracy of emotion recognition classifiers trained with CWGAN-GP-generated EEG data combined with non-EEG sensory data was compared against those trained using a combination of real EEG and non-EEG sensory data, the accuracy ratios were 93% on the SEED-V dataset, 99% on the DEAP dataset, and 97% on the GraffitiVR dataset. Moreover, in the GraffitiVR dataset, using CWGAN-GP-generated EEG data with non-EEG sensory data for emotion recognition models resulted in up to a 30% reduction in calibration time compared to classifiers trained on real EEG data with non-EEG sensory data. These results underscore the robustness and versatility of the proposed approach, significantly enhancing emotion recognition processes across a variety of environmental settings.https://peerj.com/articles/cs-2649.pdfBrain-computer interfaceCalibrationEmotion recognitionGenerative adversarial networkConditional Wasserstein GANSynthetic EEG generation |
| spellingShingle | Mehmet Ali Sarikaya Gökhan Ince Improved BCI calibration in multimodal emotion recognition using heterogeneous adversarial transfer learning PeerJ Computer Science Brain-computer interface Calibration Emotion recognition Generative adversarial network Conditional Wasserstein GAN Synthetic EEG generation |
| title | Improved BCI calibration in multimodal emotion recognition using heterogeneous adversarial transfer learning |
| title_full | Improved BCI calibration in multimodal emotion recognition using heterogeneous adversarial transfer learning |
| title_fullStr | Improved BCI calibration in multimodal emotion recognition using heterogeneous adversarial transfer learning |
| title_full_unstemmed | Improved BCI calibration in multimodal emotion recognition using heterogeneous adversarial transfer learning |
| title_short | Improved BCI calibration in multimodal emotion recognition using heterogeneous adversarial transfer learning |
| title_sort | improved bci calibration in multimodal emotion recognition using heterogeneous adversarial transfer learning |
| topic | Brain-computer interface Calibration Emotion recognition Generative adversarial network Conditional Wasserstein GAN Synthetic EEG generation |
| url | https://peerj.com/articles/cs-2649.pdf |
| work_keys_str_mv | AT mehmetalisarikaya improvedbcicalibrationinmultimodalemotionrecognitionusingheterogeneousadversarialtransferlearning AT gokhanince improvedbcicalibrationinmultimodalemotionrecognitionusingheterogeneousadversarialtransferlearning |