HGTFM: Hierarchical Gating-Driven Transformer Fusion Model for Robust Multimodal Sentiment Analysis

HGTFM: Hierarchical Gating-Driven Transformer Fusion Model for Robust Multimodal Sentiment Analysis

In multimodal sentiment analysis, a significant challenge lies in quantifying the contribution of each modality and achieving effective modality fusion. This paper presents a Hierarchical Gating-Driven Transformer Fusion Model (HGTFM), which effectively achieves multimodal data fusion through an adv...

Full description

Saved in:
Bibliographic Details
Main Authors: Chengcheng Yang, Zhiyao Liang, Dashun Yan, Zeng Hu, Ting Wu
Format: Article
Language:English
Published: IEEE 2025-01-01
Series:IEEE Access
Subjects:
Multimodal sentiment analysis
hierarchical gated fusion module
modality interaction network
unimodal label generation module
Online Access:https://ieeexplore.ieee.org/document/10965686/
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In multimodal sentiment analysis, a significant challenge lies in quantifying the contribution of each modality and achieving effective modality fusion. This paper presents a Hierarchical Gating-Driven Transformer Fusion Model (HGTFM), which effectively achieves multimodal data fusion through an advanced transformer architecture and a Hierarchical Gated Fusion Module (HGFM). The model first encodes audio and video features using LSTM and CNN, respectively, while employing BERT to encode text features. This approach effectively enhances the model ability to manage long-range temporal dependencies. Furthermore, the model introduces HGFM, which dynamically integrates multimodal features across different levels after processing by the modality interaction network (MIN), thus intelligently adjusting the contribution weights of each modality. In particular, the model incorporates a Unimodal Label Generation Module (ULGM), which considers the differences between the modalities. This enables HGFM to more accurately identify the contributions of samples with significant modal variations, thus optimizing the module potential. The experimental results on several established benchmark datasets demonstrate that HGTFM achieves comparable performance improvements in multimodal sentiment analysis tasks.
ISSN:2169-3536

Similar Items