SSDFusion: A Semantic Segmentation Driven Framework for Infrared and Visible Image Fusion
Fusing infrared images with visible images facilitates obtaining more abundant and accurate information content. However, existing infrared and visible image fusion methods often lack attention to the semantic information and global context information in the original images. To address these issues...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/11029297/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849421861835767808 |
|---|---|
| author | Qishen Lv Rui Yang Chengmin Zhang Shuaihui Liu Xinyan Fan Zihao Luo |
| author_facet | Qishen Lv Rui Yang Chengmin Zhang Shuaihui Liu Xinyan Fan Zihao Luo |
| author_sort | Qishen Lv |
| collection | DOAJ |
| description | Fusing infrared images with visible images facilitates obtaining more abundant and accurate information content. However, existing infrared and visible image fusion methods often lack attention to the semantic information and global context information in the original images. To address these issues, we propose a novel deep learning framework for infrared and visible image fusion, which is named Semantic Segmentation Driven Infrared and Visible Image Fusion Framework (SSDFusion). Within the fusion framework, the Local Global Feature Extraction Fusion Module is employed, complemented by the decoder. Furthermore, under the guidance of semantic segmentation, SSDFusion achieves a better understanding of complex scene region information, enhancing fusion task performance. Finally, an adaptive loss function is implemented throughout SSDFusion to fine-tune the balance between the semantic segmentation task and the image fusion task by adjusting their proportional contributions. This approach aids in more accurately preserving the semantic information in the image, thereby enhancing the performance of the fusion framework. We conducted comparative experiments on the MSRS dataset with existing advanced fusion methods. The experimental results show that SSDFusion performs best in both qualitative and quantitative metrics. Analysis of the public datasets indicates that our algorithm can improve the entropy (EN), spatial frequency (SF), standard deviation (SD), mutual information (MI), visual information fidelity (VIF), and edge-based similarity measure (Q<inline-formula> <tex-math notation="LaTeX">${}_{\text {AB/F}}$ </tex-math></inline-formula>) metrics with about 15.33%, 91.55%, 17.09%, 93.39%, 66.94%, and 122.56% gains, respectively. The ablation study further demonstrates that the local global feature fusion module, the adaptive fusion loss function, and the integration of semantic segmentation and image fusion have significant effects on improving the model performance. SSDFusion also exhibits excellent performance in terms of computational efficiency and parameter count. Furthermore, we have also verified the good generalization ability of SSDFusion on the RoadScene and M3FD datasets. |
| format | Article |
| id | doaj-art-01ee3303c7514c25a12f5bf928e63039 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-01ee3303c7514c25a12f5bf928e630392025-08-20T03:31:21ZengIEEEIEEE Access2169-35362025-01-011310235910237210.1109/ACCESS.2025.357847811029297SSDFusion: A Semantic Segmentation Driven Framework for Infrared and Visible Image FusionQishen Lv0https://orcid.org/0009-0001-1917-5033Rui Yang1https://orcid.org/0009-0006-8592-7375Chengmin Zhang2Shuaihui Liu3Xinyan Fan4Zihao Luo5School of Electronic Engineering, Jiangsu Ocean University, Lianyungang, Jiangsu, ChinaSchool of Electronic Engineering, Jiangsu Ocean University, Lianyungang, Jiangsu, ChinaSchool of Electronic Engineering, Jiangsu Ocean University, Lianyungang, Jiangsu, ChinaSchool of Electronic Engineering, Jiangsu Ocean University, Lianyungang, Jiangsu, ChinaSchool of Electronic Engineering, Jiangsu Ocean University, Lianyungang, Jiangsu, ChinaSchool of Electronic Engineering, Jiangsu Ocean University, Lianyungang, Jiangsu, ChinaFusing infrared images with visible images facilitates obtaining more abundant and accurate information content. However, existing infrared and visible image fusion methods often lack attention to the semantic information and global context information in the original images. To address these issues, we propose a novel deep learning framework for infrared and visible image fusion, which is named Semantic Segmentation Driven Infrared and Visible Image Fusion Framework (SSDFusion). Within the fusion framework, the Local Global Feature Extraction Fusion Module is employed, complemented by the decoder. Furthermore, under the guidance of semantic segmentation, SSDFusion achieves a better understanding of complex scene region information, enhancing fusion task performance. Finally, an adaptive loss function is implemented throughout SSDFusion to fine-tune the balance between the semantic segmentation task and the image fusion task by adjusting their proportional contributions. This approach aids in more accurately preserving the semantic information in the image, thereby enhancing the performance of the fusion framework. We conducted comparative experiments on the MSRS dataset with existing advanced fusion methods. The experimental results show that SSDFusion performs best in both qualitative and quantitative metrics. Analysis of the public datasets indicates that our algorithm can improve the entropy (EN), spatial frequency (SF), standard deviation (SD), mutual information (MI), visual information fidelity (VIF), and edge-based similarity measure (Q<inline-formula> <tex-math notation="LaTeX">${}_{\text {AB/F}}$ </tex-math></inline-formula>) metrics with about 15.33%, 91.55%, 17.09%, 93.39%, 66.94%, and 122.56% gains, respectively. The ablation study further demonstrates that the local global feature fusion module, the adaptive fusion loss function, and the integration of semantic segmentation and image fusion have significant effects on improving the model performance. SSDFusion also exhibits excellent performance in terms of computational efficiency and parameter count. Furthermore, we have also verified the good generalization ability of SSDFusion on the RoadScene and M3FD datasets.https://ieeexplore.ieee.org/document/11029297/Attention mechanismconvolutional neural networkimage fusionsemantic segmentationtransformer |
| spellingShingle | Qishen Lv Rui Yang Chengmin Zhang Shuaihui Liu Xinyan Fan Zihao Luo SSDFusion: A Semantic Segmentation Driven Framework for Infrared and Visible Image Fusion IEEE Access Attention mechanism convolutional neural network image fusion semantic segmentation transformer |
| title | SSDFusion: A Semantic Segmentation Driven Framework for Infrared and Visible Image Fusion |
| title_full | SSDFusion: A Semantic Segmentation Driven Framework for Infrared and Visible Image Fusion |
| title_fullStr | SSDFusion: A Semantic Segmentation Driven Framework for Infrared and Visible Image Fusion |
| title_full_unstemmed | SSDFusion: A Semantic Segmentation Driven Framework for Infrared and Visible Image Fusion |
| title_short | SSDFusion: A Semantic Segmentation Driven Framework for Infrared and Visible Image Fusion |
| title_sort | ssdfusion a semantic segmentation driven framework for infrared and visible image fusion |
| topic | Attention mechanism convolutional neural network image fusion semantic segmentation transformer |
| url | https://ieeexplore.ieee.org/document/11029297/ |
| work_keys_str_mv | AT qishenlv ssdfusionasemanticsegmentationdrivenframeworkforinfraredandvisibleimagefusion AT ruiyang ssdfusionasemanticsegmentationdrivenframeworkforinfraredandvisibleimagefusion AT chengminzhang ssdfusionasemanticsegmentationdrivenframeworkforinfraredandvisibleimagefusion AT shuaihuiliu ssdfusionasemanticsegmentationdrivenframeworkforinfraredandvisibleimagefusion AT xinyanfan ssdfusionasemanticsegmentationdrivenframeworkforinfraredandvisibleimagefusion AT zihaoluo ssdfusionasemanticsegmentationdrivenframeworkforinfraredandvisibleimagefusion |