KE-RSIC: Remote Sensing Image Captioning Based on Knowledge Embedding
Current remote sensing image captioning methods often struggle to provide accurate and comprehensive descriptions due to their reliance on networks designed for natural images. Due to limited domain-specific knowledge in remote sensing, these networks often fail to accurately reflect the intrinsic s...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10818406/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832576791864147968 |
|---|---|
| author | Kangda Cheng Erik Cambria Jinlong Liu Yushi Chen Zhilu Wu |
| author_facet | Kangda Cheng Erik Cambria Jinlong Liu Yushi Chen Zhilu Wu |
| author_sort | Kangda Cheng |
| collection | DOAJ |
| description | Current remote sensing image captioning methods often struggle to provide accurate and comprehensive descriptions due to their reliance on networks designed for natural images. Due to limited domain-specific knowledge in remote sensing, these networks often fail to accurately reflect the intrinsic semantic information of remote sensing categories. This article proposes a novel knowledge-embedded remote sensing image captioning model. We first define two types of remote sensing knowledge: general knowledge within the field of remote sensing, and specific knowledge that is relevant to the input image. To acquire general knowledge, we construct a remote sensing knowledge graph and propose a general knowledge embedding method, enabling semantic correlations between entities and relationships in remote sensing knowledge graphs. The generated entity embeddings and relationship embeddings can effectively capture the intrinsic semantic information of remote sensing categories. To acquire specific knowledge, we also propose a specific knowledge embedding method. We retrieve reports with similar label distributions to the input and then extract entities and relationships from the retrieved reports using a relation extractor. Embedding specific knowledge can alleviate to some extent the issue of poor matching between visual features and semantic features due to the lack of relevant knowledge. Subsequently, to integrate entity embeddings, relationship embeddings, and visual features, we propose a visual feature and knowledge information dynamic fusion module. This module can efficiently combine the visual features of remote sensing images with structural information on embedded knowledge. Numerous experimental findings attest to the superiority and effectiveness of the proposed method. |
| format | Article |
| id | doaj-art-e5f3d889c6a246ee84c0f2eaff2ccbb3 |
| institution | Kabale University |
| issn | 1939-1404 2151-1535 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj-art-e5f3d889c6a246ee84c0f2eaff2ccbb32025-01-31T00:00:21ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing1939-14042151-15352025-01-01184286430410.1109/JSTARS.2024.352394410818406KE-RSIC: Remote Sensing Image Captioning Based on Knowledge EmbeddingKangda Cheng0https://orcid.org/0000-0001-5525-6423Erik Cambria1https://orcid.org/0000-0002-3030-1280Jinlong Liu2https://orcid.org/0000-0002-0284-5029Yushi Chen3https://orcid.org/0000-0003-2421-0996Zhilu Wu4https://orcid.org/0000-0002-3402-9093School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin, ChinaSchool of Computer Science and Engineering, Nanyang Technological University, SingaporeSchool of Electronics and Information Engineering, Harbin Institute of Technology, Harbin, ChinaSchool of Electronics and Information Engineering, Harbin Institute of Technology, Harbin, ChinaSchool of Electronics and Information Engineering, Harbin Institute of Technology, Harbin, ChinaCurrent remote sensing image captioning methods often struggle to provide accurate and comprehensive descriptions due to their reliance on networks designed for natural images. Due to limited domain-specific knowledge in remote sensing, these networks often fail to accurately reflect the intrinsic semantic information of remote sensing categories. This article proposes a novel knowledge-embedded remote sensing image captioning model. We first define two types of remote sensing knowledge: general knowledge within the field of remote sensing, and specific knowledge that is relevant to the input image. To acquire general knowledge, we construct a remote sensing knowledge graph and propose a general knowledge embedding method, enabling semantic correlations between entities and relationships in remote sensing knowledge graphs. The generated entity embeddings and relationship embeddings can effectively capture the intrinsic semantic information of remote sensing categories. To acquire specific knowledge, we also propose a specific knowledge embedding method. We retrieve reports with similar label distributions to the input and then extract entities and relationships from the retrieved reports using a relation extractor. Embedding specific knowledge can alleviate to some extent the issue of poor matching between visual features and semantic features due to the lack of relevant knowledge. Subsequently, to integrate entity embeddings, relationship embeddings, and visual features, we propose a visual feature and knowledge information dynamic fusion module. This module can efficiently combine the visual features of remote sensing images with structural information on embedded knowledge. Numerous experimental findings attest to the superiority and effectiveness of the proposed method.https://ieeexplore.ieee.org/document/10818406/Image captionknowledge embeddingremote sensingrepresentation learning |
| spellingShingle | Kangda Cheng Erik Cambria Jinlong Liu Yushi Chen Zhilu Wu KE-RSIC: Remote Sensing Image Captioning Based on Knowledge Embedding IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Image caption knowledge embedding remote sensing representation learning |
| title | KE-RSIC: Remote Sensing Image Captioning Based on Knowledge Embedding |
| title_full | KE-RSIC: Remote Sensing Image Captioning Based on Knowledge Embedding |
| title_fullStr | KE-RSIC: Remote Sensing Image Captioning Based on Knowledge Embedding |
| title_full_unstemmed | KE-RSIC: Remote Sensing Image Captioning Based on Knowledge Embedding |
| title_short | KE-RSIC: Remote Sensing Image Captioning Based on Knowledge Embedding |
| title_sort | ke rsic remote sensing image captioning based on knowledge embedding |
| topic | Image caption knowledge embedding remote sensing representation learning |
| url | https://ieeexplore.ieee.org/document/10818406/ |
| work_keys_str_mv | AT kangdacheng kersicremotesensingimagecaptioningbasedonknowledgeembedding AT erikcambria kersicremotesensingimagecaptioningbasedonknowledgeembedding AT jinlongliu kersicremotesensingimagecaptioningbasedonknowledgeembedding AT yushichen kersicremotesensingimagecaptioningbasedonknowledgeembedding AT zhiluwu kersicremotesensingimagecaptioningbasedonknowledgeembedding |