Robust Fusion of Time Series and Image Data for Improved Multimodal Clinical Prediction
With the increasing availability of diverse data types, particularly images and time series data from medical experiments, there is a growing demand for techniques designed to combine various modalities of data effectively. Our motivation comes from the important areas of predicting mortality and ph...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10752509/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850267878168723456 |
|---|---|
| author | Ali Rasekh Reza Heidari Amir Hosein Haji Mohammad Rezaie Parsa Sharifi Sedeh Zahra Ahmadi Prasenjit Mitra Wolfgang Nejdl |
| author_facet | Ali Rasekh Reza Heidari Amir Hosein Haji Mohammad Rezaie Parsa Sharifi Sedeh Zahra Ahmadi Prasenjit Mitra Wolfgang Nejdl |
| author_sort | Ali Rasekh |
| collection | DOAJ |
| description | With the increasing availability of diverse data types, particularly images and time series data from medical experiments, there is a growing demand for techniques designed to combine various modalities of data effectively. Our motivation comes from the important areas of predicting mortality and phenotyping where using different modalities of data could significantly improve our ability to predict. To tackle this challenge, we introduce a new method that uses two separate encoders, one for each type of data, allowing the model to understand complex patterns in both visual and time-based information. Apart from the technical challenges, our goal is to make the predictive model more robust in noisy conditions and perform better than current methods. We also deal with imbalanced datasets and use an uncertainty loss function, yielding improved results while simultaneously providing a principled means of modeling uncertainty. Additionally, we include attention mechanisms to fuse different modalities, allowing the model to focus on what’s important for each task. We tested our approach using the comprehensive multimodal MIMIC dataset, combining MIMIC-IV and MIMIC-CXR datasets. Our experiments show that our method is effective in improving multimodal deep learning for clinical applications. The code for this work is publicly available at: <uri>https://github.com/AliRasekh/TSImageFusion</uri> |
| format | Article |
| id | doaj-art-ff7d69c3aead48a3aad623b80c2bca03 |
| institution | OA Journals |
| issn | 2169-3536 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-ff7d69c3aead48a3aad623b80c2bca032025-08-20T01:53:36ZengIEEEIEEE Access2169-35362024-01-011217410717412110.1109/ACCESS.2024.349766810752509Robust Fusion of Time Series and Image Data for Improved Multimodal Clinical PredictionAli Rasekh0https://orcid.org/0009-0005-6808-6730Reza Heidari1https://orcid.org/0009-0007-6260-6592Amir Hosein Haji Mohammad Rezaie2https://orcid.org/0009-0008-5072-971XParsa Sharifi Sedeh3https://orcid.org/0009-0000-3621-1887Zahra Ahmadi4Prasenjit Mitra5Wolfgang Nejdl6L3S Research Center, Leibniz University Hannover, Hannover, GermanyL3S Research Center, Leibniz University Hannover, Hannover, GermanyL3S Research Center, Leibniz University Hannover, Hannover, GermanyL3S Research Center, Leibniz University Hannover, Hannover, GermanyL3S Research Center, Leibniz University Hannover, Hannover, GermanyL3S Research Center, Leibniz University Hannover, Hannover, GermanyL3S Research Center, Leibniz University Hannover, Hannover, GermanyWith the increasing availability of diverse data types, particularly images and time series data from medical experiments, there is a growing demand for techniques designed to combine various modalities of data effectively. Our motivation comes from the important areas of predicting mortality and phenotyping where using different modalities of data could significantly improve our ability to predict. To tackle this challenge, we introduce a new method that uses two separate encoders, one for each type of data, allowing the model to understand complex patterns in both visual and time-based information. Apart from the technical challenges, our goal is to make the predictive model more robust in noisy conditions and perform better than current methods. We also deal with imbalanced datasets and use an uncertainty loss function, yielding improved results while simultaneously providing a principled means of modeling uncertainty. Additionally, we include attention mechanisms to fuse different modalities, allowing the model to focus on what’s important for each task. We tested our approach using the comprehensive multimodal MIMIC dataset, combining MIMIC-IV and MIMIC-CXR datasets. Our experiments show that our method is effective in improving multimodal deep learning for clinical applications. The code for this work is publicly available at: <uri>https://github.com/AliRasekh/TSImageFusion</uri>https://ieeexplore.ieee.org/document/10752509/Multimodal learningtime seriesattention mechanismrobustnessphenotyping |
| spellingShingle | Ali Rasekh Reza Heidari Amir Hosein Haji Mohammad Rezaie Parsa Sharifi Sedeh Zahra Ahmadi Prasenjit Mitra Wolfgang Nejdl Robust Fusion of Time Series and Image Data for Improved Multimodal Clinical Prediction IEEE Access Multimodal learning time series attention mechanism robustness phenotyping |
| title | Robust Fusion of Time Series and Image Data for Improved Multimodal Clinical Prediction |
| title_full | Robust Fusion of Time Series and Image Data for Improved Multimodal Clinical Prediction |
| title_fullStr | Robust Fusion of Time Series and Image Data for Improved Multimodal Clinical Prediction |
| title_full_unstemmed | Robust Fusion of Time Series and Image Data for Improved Multimodal Clinical Prediction |
| title_short | Robust Fusion of Time Series and Image Data for Improved Multimodal Clinical Prediction |
| title_sort | robust fusion of time series and image data for improved multimodal clinical prediction |
| topic | Multimodal learning time series attention mechanism robustness phenotyping |
| url | https://ieeexplore.ieee.org/document/10752509/ |
| work_keys_str_mv | AT alirasekh robustfusionoftimeseriesandimagedataforimprovedmultimodalclinicalprediction AT rezaheidari robustfusionoftimeseriesandimagedataforimprovedmultimodalclinicalprediction AT amirhoseinhajimohammadrezaie robustfusionoftimeseriesandimagedataforimprovedmultimodalclinicalprediction AT parsasharifisedeh robustfusionoftimeseriesandimagedataforimprovedmultimodalclinicalprediction AT zahraahmadi robustfusionoftimeseriesandimagedataforimprovedmultimodalclinicalprediction AT prasenjitmitra robustfusionoftimeseriesandimagedataforimprovedmultimodalclinicalprediction AT wolfgangnejdl robustfusionoftimeseriesandimagedataforimprovedmultimodalclinicalprediction |