Face and Voice Recognition-Based Emotion Analysis System (EAS) to Minimize Heterogeneity in the Metaverse
The metaverse, where users interact through avatars, is evolving to closely mirror the real world, requiring realistic object responses based on users’ emotions. While technologies like eye-tracking and hand-tracking transfer physical movements into virtual spaces, accurate emotion detection remains...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-01-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/2/845 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832589163051876352 |
|---|---|
| author | Surak Son Yina Jeong |
| author_facet | Surak Son Yina Jeong |
| author_sort | Surak Son |
| collection | DOAJ |
| description | The metaverse, where users interact through avatars, is evolving to closely mirror the real world, requiring realistic object responses based on users’ emotions. While technologies like eye-tracking and hand-tracking transfer physical movements into virtual spaces, accurate emotion detection remains challenging. This study proposes the “Face and Voice Recognition-based Emotion Analysis System (EAS)” to bridge this gap, assessing emotions through both voice and facial expressions. EAS utilizes a microphone and camera to gauge emotional states, combining these inputs for a comprehensive analysis. It comprises three neural networks: the Facial Emotion Analysis Model (FEAM), which classifies emotions using facial landmarks; the Voice Sentiment Analysis Model (VSAM), which detects vocal emotions even in noisy environments using MCycleGAN; and the Metaverse Emotion Recognition Model (MERM), which integrates FEAM and VSAM outputs to infer overall emotional states. EAS’s three primary modules—Facial Emotion Recognition, Voice Emotion Recognition, and User Emotion Analysis—analyze facial features and vocal tones to detect emotions, providing a holistic emotional assessment for realistic interactions in the metaverse. The system’s performance is validated through dataset testing, and future directions are suggested based on simulation outcomes. |
| format | Article |
| id | doaj-art-099e7111baee467da011fb306d6a0fae |
| institution | Kabale University |
| issn | 2076-3417 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-099e7111baee467da011fb306d6a0fae2025-01-24T13:21:02ZengMDPI AGApplied Sciences2076-34172025-01-0115284510.3390/app15020845Face and Voice Recognition-Based Emotion Analysis System (EAS) to Minimize Heterogeneity in the MetaverseSurak Son0Yina Jeong1Department of Software, College of Engineering, Catholic Kwandong University, Gangneung 25601, Republic of KoreaDepartment of Software, College of Engineering, Catholic Kwandong University, Gangneung 25601, Republic of KoreaThe metaverse, where users interact through avatars, is evolving to closely mirror the real world, requiring realistic object responses based on users’ emotions. While technologies like eye-tracking and hand-tracking transfer physical movements into virtual spaces, accurate emotion detection remains challenging. This study proposes the “Face and Voice Recognition-based Emotion Analysis System (EAS)” to bridge this gap, assessing emotions through both voice and facial expressions. EAS utilizes a microphone and camera to gauge emotional states, combining these inputs for a comprehensive analysis. It comprises three neural networks: the Facial Emotion Analysis Model (FEAM), which classifies emotions using facial landmarks; the Voice Sentiment Analysis Model (VSAM), which detects vocal emotions even in noisy environments using MCycleGAN; and the Metaverse Emotion Recognition Model (MERM), which integrates FEAM and VSAM outputs to infer overall emotional states. EAS’s three primary modules—Facial Emotion Recognition, Voice Emotion Recognition, and User Emotion Analysis—analyze facial features and vocal tones to detect emotions, providing a holistic emotional assessment for realistic interactions in the metaverse. The system’s performance is validated through dataset testing, and future directions are suggested based on simulation outcomes.https://www.mdpi.com/2076-3417/15/2/845artificial intelligenceemotion recognitionfacial expression analysisvoice analysisfacial-voice matching |
| spellingShingle | Surak Son Yina Jeong Face and Voice Recognition-Based Emotion Analysis System (EAS) to Minimize Heterogeneity in the Metaverse Applied Sciences artificial intelligence emotion recognition facial expression analysis voice analysis facial-voice matching |
| title | Face and Voice Recognition-Based Emotion Analysis System (EAS) to Minimize Heterogeneity in the Metaverse |
| title_full | Face and Voice Recognition-Based Emotion Analysis System (EAS) to Minimize Heterogeneity in the Metaverse |
| title_fullStr | Face and Voice Recognition-Based Emotion Analysis System (EAS) to Minimize Heterogeneity in the Metaverse |
| title_full_unstemmed | Face and Voice Recognition-Based Emotion Analysis System (EAS) to Minimize Heterogeneity in the Metaverse |
| title_short | Face and Voice Recognition-Based Emotion Analysis System (EAS) to Minimize Heterogeneity in the Metaverse |
| title_sort | face and voice recognition based emotion analysis system eas to minimize heterogeneity in the metaverse |
| topic | artificial intelligence emotion recognition facial expression analysis voice analysis facial-voice matching |
| url | https://www.mdpi.com/2076-3417/15/2/845 |
| work_keys_str_mv | AT surakson faceandvoicerecognitionbasedemotionanalysissystemeastominimizeheterogeneityinthemetaverse AT yinajeong faceandvoicerecognitionbasedemotionanalysissystemeastominimizeheterogeneityinthemetaverse |