Open-world disaster information identification from multimodal social media
Abstract The application of multimodal deep learning for emergency response and recovery, specifically in disaster social media analysis, is of utmost importance. It is worth noting that in real-world scenarios, sudden disaster events may differ from the training data, which may require the multimod...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Springer
2024-11-01
|
| Series: | Complex & Intelligent Systems |
| Subjects: | |
| Online Access: | https://doi.org/10.1007/s40747-024-01635-5 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832571182511030272 |
|---|---|
| author | Chen Yu Bin Hu Zhiguo Wang |
| author_facet | Chen Yu Bin Hu Zhiguo Wang |
| author_sort | Chen Yu |
| collection | DOAJ |
| description | Abstract The application of multimodal deep learning for emergency response and recovery, specifically in disaster social media analysis, is of utmost importance. It is worth noting that in real-world scenarios, sudden disaster events may differ from the training data, which may require the multimodal network to predict them as unknown classes instead of misclassifying them to known ones. Previous studies have primarily focused on model accuracy in a closed environment and may not be able to directly detect unknown classes. Thus, we propose a novel multimodal model for categorizing social media related to disasters in an open-world environment. Our methodology entails utilizing pre-trained unimodal models as encoders for each modality and performing information fusion with a cross-attention module to obtain the joint representation. For open-world detection, we use a multitask classifier that encompasses both a closed-world and an open-world classifier. The closed-world classifier is trained on the original data to classify known classes, whereas the open-world classifier is used to determine whether the input belongs to a known class. Furthermore, we propose a sample generation strategy that models the distribution of unknown samples using known data, which allows the open-world classifier to identify unknown samples. Our experiments were conducted on two public datasets, namely CrisisMMD and MHII. According to the experimental results, the proposed method outperforms other baselines and approaches in crisis information classification. |
| format | Article |
| id | doaj-art-903bcbaf7de445898132d678d5aaf47e |
| institution | Kabale University |
| issn | 2199-4536 2198-6053 |
| language | English |
| publishDate | 2024-11-01 |
| publisher | Springer |
| record_format | Article |
| series | Complex & Intelligent Systems |
| spelling | doaj-art-903bcbaf7de445898132d678d5aaf47e2025-02-02T12:50:02ZengSpringerComplex & Intelligent Systems2199-45362198-60532024-11-0111111310.1007/s40747-024-01635-5Open-world disaster information identification from multimodal social mediaChen Yu0Bin Hu1Zhiguo Wang2University of Electronic Science and Technology of ChinaChengdu Institute of the Tibetan Plateau Earthquake Research, China Earthquake AdministrationUniversity of Electronic Science and Technology of ChinaAbstract The application of multimodal deep learning for emergency response and recovery, specifically in disaster social media analysis, is of utmost importance. It is worth noting that in real-world scenarios, sudden disaster events may differ from the training data, which may require the multimodal network to predict them as unknown classes instead of misclassifying them to known ones. Previous studies have primarily focused on model accuracy in a closed environment and may not be able to directly detect unknown classes. Thus, we propose a novel multimodal model for categorizing social media related to disasters in an open-world environment. Our methodology entails utilizing pre-trained unimodal models as encoders for each modality and performing information fusion with a cross-attention module to obtain the joint representation. For open-world detection, we use a multitask classifier that encompasses both a closed-world and an open-world classifier. The closed-world classifier is trained on the original data to classify known classes, whereas the open-world classifier is used to determine whether the input belongs to a known class. Furthermore, we propose a sample generation strategy that models the distribution of unknown samples using known data, which allows the open-world classifier to identify unknown samples. Our experiments were conducted on two public datasets, namely CrisisMMD and MHII. According to the experimental results, the proposed method outperforms other baselines and approaches in crisis information classification.https://doi.org/10.1007/s40747-024-01635-5Open-world learningDisaster tweet classificationMultimodal learningEmergency responseTransformer |
| spellingShingle | Chen Yu Bin Hu Zhiguo Wang Open-world disaster information identification from multimodal social media Complex & Intelligent Systems Open-world learning Disaster tweet classification Multimodal learning Emergency response Transformer |
| title | Open-world disaster information identification from multimodal social media |
| title_full | Open-world disaster information identification from multimodal social media |
| title_fullStr | Open-world disaster information identification from multimodal social media |
| title_full_unstemmed | Open-world disaster information identification from multimodal social media |
| title_short | Open-world disaster information identification from multimodal social media |
| title_sort | open world disaster information identification from multimodal social media |
| topic | Open-world learning Disaster tweet classification Multimodal learning Emergency response Transformer |
| url | https://doi.org/10.1007/s40747-024-01635-5 |
| work_keys_str_mv | AT chenyu openworlddisasterinformationidentificationfrommultimodalsocialmedia AT binhu openworlddisasterinformationidentificationfrommultimodalsocialmedia AT zhiguowang openworlddisasterinformationidentificationfrommultimodalsocialmedia |