Channel Mixer Layer: Multimodal Fusion Toward Machine Reasoning for Spatiotemporal Predictive Learning of Ionospheric Total Electron Content
Abstract The spatiotemporal distribution of Total Electron Content (TEC) in ionosphere determines the refractive index of electromagnetic wave leading to the radio signal scintillation and deterioration. Thanks to the development of machine learning for video prediction, spatiotemporal predictive mo...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2024-12-01
|
| Series: | Space Weather |
| Subjects: | |
| Online Access: | https://doi.org/10.1029/2024SW004121 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832575415457153024 |
|---|---|
| author | Peng Liu Tatsuhiro Yokoyama Takuya Sori Mamoru Yamamoto |
| author_facet | Peng Liu Tatsuhiro Yokoyama Takuya Sori Mamoru Yamamoto |
| author_sort | Peng Liu |
| collection | DOAJ |
| description | Abstract The spatiotemporal distribution of Total Electron Content (TEC) in ionosphere determines the refractive index of electromagnetic wave leading to the radio signal scintillation and deterioration. Thanks to the development of machine learning for video prediction, spatiotemporal predictive models are applied on the future TEC map prediction based on the graphic features of past frames. However, output result of graphic prediction is unable to properly respond to the external factor variations such as solar or geomagnetic activity. Meanwhile, there is still neither standard data ‐set nor comprehensive evaluation framework for spatiotemporal predictive learning of TEC map sequences leading to the comparisons unfair and insights inconclusive. In this research, a new feature‐level multimodal fusion method named as channel mixer layer for machine reasoning is proposed that can be embedded into the existing advanced spatiotemporal sequence prediction models. Meanwhile, all performance benchmarks are accomplished on the same running environment and newly proposed largest scale data set. Experiment results suggest that the multimodal fusion prediction of existing model backbones by proposed method improves the prediction accuracy up to 15% with almost the same computational complexity compared to that of graphic prediction without auxiliary factors input, having the real‐time inference speed of 34 frames/second and minimum mean absolute error of 0.94/2.63 TEC unit during low/high solar activity period respectively. The channel mixer layer embedded models can respond to the variations of auxiliary external factors more correctly than previous multimodal fusion methods such as concatenation and arithmetic, which is regarded as the evidence of state‐of‐the‐art machine reasoning ability. |
| format | Article |
| id | doaj-art-2d282ff0a0784e8a806a1460b46db348 |
| institution | Kabale University |
| issn | 1542-7390 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | Wiley |
| record_format | Article |
| series | Space Weather |
| spelling | doaj-art-2d282ff0a0784e8a806a1460b46db3482025-02-01T08:10:32ZengWileySpace Weather1542-73902024-12-012212n/an/a10.1029/2024SW004121Channel Mixer Layer: Multimodal Fusion Toward Machine Reasoning for Spatiotemporal Predictive Learning of Ionospheric Total Electron ContentPeng Liu0Tatsuhiro Yokoyama1Takuya Sori2Mamoru Yamamoto3Research Institute for Sustainable Humanosphere Kyoto University Uji City JapanResearch Institute for Sustainable Humanosphere Kyoto University Uji City JapanResearch Institute for Sustainable Humanosphere Kyoto University Uji City JapanResearch Institute for Sustainable Humanosphere Kyoto University Uji City JapanAbstract The spatiotemporal distribution of Total Electron Content (TEC) in ionosphere determines the refractive index of electromagnetic wave leading to the radio signal scintillation and deterioration. Thanks to the development of machine learning for video prediction, spatiotemporal predictive models are applied on the future TEC map prediction based on the graphic features of past frames. However, output result of graphic prediction is unable to properly respond to the external factor variations such as solar or geomagnetic activity. Meanwhile, there is still neither standard data ‐set nor comprehensive evaluation framework for spatiotemporal predictive learning of TEC map sequences leading to the comparisons unfair and insights inconclusive. In this research, a new feature‐level multimodal fusion method named as channel mixer layer for machine reasoning is proposed that can be embedded into the existing advanced spatiotemporal sequence prediction models. Meanwhile, all performance benchmarks are accomplished on the same running environment and newly proposed largest scale data set. Experiment results suggest that the multimodal fusion prediction of existing model backbones by proposed method improves the prediction accuracy up to 15% with almost the same computational complexity compared to that of graphic prediction without auxiliary factors input, having the real‐time inference speed of 34 frames/second and minimum mean absolute error of 0.94/2.63 TEC unit during low/high solar activity period respectively. The channel mixer layer embedded models can respond to the variations of auxiliary external factors more correctly than previous multimodal fusion methods such as concatenation and arithmetic, which is regarded as the evidence of state‐of‐the‐art machine reasoning ability.https://doi.org/10.1029/2024SW004121multimodal fusionmachine reasoningspatiotemporal predictive learningionosphereTotal Electron Contentdeep learning |
| spellingShingle | Peng Liu Tatsuhiro Yokoyama Takuya Sori Mamoru Yamamoto Channel Mixer Layer: Multimodal Fusion Toward Machine Reasoning for Spatiotemporal Predictive Learning of Ionospheric Total Electron Content Space Weather multimodal fusion machine reasoning spatiotemporal predictive learning ionosphere Total Electron Content deep learning |
| title | Channel Mixer Layer: Multimodal Fusion Toward Machine Reasoning for Spatiotemporal Predictive Learning of Ionospheric Total Electron Content |
| title_full | Channel Mixer Layer: Multimodal Fusion Toward Machine Reasoning for Spatiotemporal Predictive Learning of Ionospheric Total Electron Content |
| title_fullStr | Channel Mixer Layer: Multimodal Fusion Toward Machine Reasoning for Spatiotemporal Predictive Learning of Ionospheric Total Electron Content |
| title_full_unstemmed | Channel Mixer Layer: Multimodal Fusion Toward Machine Reasoning for Spatiotemporal Predictive Learning of Ionospheric Total Electron Content |
| title_short | Channel Mixer Layer: Multimodal Fusion Toward Machine Reasoning for Spatiotemporal Predictive Learning of Ionospheric Total Electron Content |
| title_sort | channel mixer layer multimodal fusion toward machine reasoning for spatiotemporal predictive learning of ionospheric total electron content |
| topic | multimodal fusion machine reasoning spatiotemporal predictive learning ionosphere Total Electron Content deep learning |
| url | https://doi.org/10.1029/2024SW004121 |
| work_keys_str_mv | AT pengliu channelmixerlayermultimodalfusiontowardmachinereasoningforspatiotemporalpredictivelearningofionospherictotalelectroncontent AT tatsuhiroyokoyama channelmixerlayermultimodalfusiontowardmachinereasoningforspatiotemporalpredictivelearningofionospherictotalelectroncontent AT takuyasori channelmixerlayermultimodalfusiontowardmachinereasoningforspatiotemporalpredictivelearningofionospherictotalelectroncontent AT mamoruyamamoto channelmixerlayermultimodalfusiontowardmachinereasoningforspatiotemporalpredictivelearningofionospherictotalelectroncontent |