A Social Media Dataset and H-GNN-Based Contrastive Learning Scheme for Multimodal Sentiment Analysis
Multimodal sentiment analysis faces a number of challenges, including modality missing, modality heterogeneity gap, incomplete datasets, etc. Previous studies usually adopt schemes like meta-learning or multi-layer structures. Nevertheless, these methods lack interpretability for the interaction bet...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-01-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/2/636 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832589255126286336 |
|---|---|
| author | Jiao Peng Yue He Yongjuan Chang Yanyan Lu Pengfei Zhang Zhonghong Ou Qingzhi Yu |
| author_facet | Jiao Peng Yue He Yongjuan Chang Yanyan Lu Pengfei Zhang Zhonghong Ou Qingzhi Yu |
| author_sort | Jiao Peng |
| collection | DOAJ |
| description | Multimodal sentiment analysis faces a number of challenges, including modality missing, modality heterogeneity gap, incomplete datasets, etc. Previous studies usually adopt schemes like meta-learning or multi-layer structures. Nevertheless, these methods lack interpretability for the interaction between modalities. In this paper, we constructed a new dataset, SM-MSD, for sentiment analysis in social media (SAS) that differs significantly from conventional corpora, comprising 10K instances of diverse data from Twitter, encompassing text, emoticons, emojis, and text embedded in images. This dataset aims to reflect authentic social scenarios and various emotional expressions, and provides a meaningful and challenging evaluation benchmark for multimodal sentiment analysis in specific contexts. Furthermore, we propose a multi-task framework based on heterogeneous graph neural networks (H-GNNs) and contrastive learning. For the first time, heterogeneous graph neural networks are applied to multimodal sentiment analysis tasks. In the case of additional labeling data, it guides the emotion prediction of the missing mode. We conduct extensive experiments on multiple datasets to verify the effectiveness of the proposed scheme. Experimental results demonstrate that our proposed scheme surpasses state-of-the-art methods by 1.7% and 0 in accuracy and 1.54% and 4.9% in F1-score on the MOSI and MOSEI datasets, respectively, and exhibits robustness to modality missing scenarios. |
| format | Article |
| id | doaj-art-4aeee671dda648d7a005e60058ff3bb8 |
| institution | Kabale University |
| issn | 2076-3417 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-4aeee671dda648d7a005e60058ff3bb82025-01-24T13:20:13ZengMDPI AGApplied Sciences2076-34172025-01-0115263610.3390/app15020636A Social Media Dataset and H-GNN-Based Contrastive Learning Scheme for Multimodal Sentiment AnalysisJiao Peng0Yue He1Yongjuan Chang2Yanyan Lu3Pengfei Zhang4Zhonghong Ou5Qingzhi Yu6State Grid Hebei Information and Telecommunication Branch, Shijiazhuang 050051, ChinaState Grid Hebei Information and Telecommunication Branch, Shijiazhuang 050051, ChinaState Grid Hebei Information and Telecommunication Branch, Shijiazhuang 050051, ChinaState Grid Hebei Information and Telecommunication Branch, Shijiazhuang 050051, ChinaState Grid Hebei Information and Telecommunication Branch, Shijiazhuang 050051, ChinaState Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, ChinaSchool of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing 100876, ChinaMultimodal sentiment analysis faces a number of challenges, including modality missing, modality heterogeneity gap, incomplete datasets, etc. Previous studies usually adopt schemes like meta-learning or multi-layer structures. Nevertheless, these methods lack interpretability for the interaction between modalities. In this paper, we constructed a new dataset, SM-MSD, for sentiment analysis in social media (SAS) that differs significantly from conventional corpora, comprising 10K instances of diverse data from Twitter, encompassing text, emoticons, emojis, and text embedded in images. This dataset aims to reflect authentic social scenarios and various emotional expressions, and provides a meaningful and challenging evaluation benchmark for multimodal sentiment analysis in specific contexts. Furthermore, we propose a multi-task framework based on heterogeneous graph neural networks (H-GNNs) and contrastive learning. For the first time, heterogeneous graph neural networks are applied to multimodal sentiment analysis tasks. In the case of additional labeling data, it guides the emotion prediction of the missing mode. We conduct extensive experiments on multiple datasets to verify the effectiveness of the proposed scheme. Experimental results demonstrate that our proposed scheme surpasses state-of-the-art methods by 1.7% and 0 in accuracy and 1.54% and 4.9% in F1-score on the MOSI and MOSEI datasets, respectively, and exhibits robustness to modality missing scenarios.https://www.mdpi.com/2076-3417/15/2/636multimodal sentiment analysisdatasetscontrastive learningheterogeneous graph neural networks |
| spellingShingle | Jiao Peng Yue He Yongjuan Chang Yanyan Lu Pengfei Zhang Zhonghong Ou Qingzhi Yu A Social Media Dataset and H-GNN-Based Contrastive Learning Scheme for Multimodal Sentiment Analysis Applied Sciences multimodal sentiment analysis datasets contrastive learning heterogeneous graph neural networks |
| title | A Social Media Dataset and H-GNN-Based Contrastive Learning Scheme for Multimodal Sentiment Analysis |
| title_full | A Social Media Dataset and H-GNN-Based Contrastive Learning Scheme for Multimodal Sentiment Analysis |
| title_fullStr | A Social Media Dataset and H-GNN-Based Contrastive Learning Scheme for Multimodal Sentiment Analysis |
| title_full_unstemmed | A Social Media Dataset and H-GNN-Based Contrastive Learning Scheme for Multimodal Sentiment Analysis |
| title_short | A Social Media Dataset and H-GNN-Based Contrastive Learning Scheme for Multimodal Sentiment Analysis |
| title_sort | social media dataset and h gnn based contrastive learning scheme for multimodal sentiment analysis |
| topic | multimodal sentiment analysis datasets contrastive learning heterogeneous graph neural networks |
| url | https://www.mdpi.com/2076-3417/15/2/636 |
| work_keys_str_mv | AT jiaopeng asocialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT yuehe asocialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT yongjuanchang asocialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT yanyanlu asocialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT pengfeizhang asocialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT zhonghongou asocialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT qingzhiyu asocialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT jiaopeng socialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT yuehe socialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT yongjuanchang socialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT yanyanlu socialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT pengfeizhang socialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT zhonghongou socialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis AT qingzhiyu socialmediadatasetandhgnnbasedcontrastivelearningschemeformultimodalsentimentanalysis |