RSPS-SAM: A Remote Sensing Image Panoptic Segmentation Method Based on SAM
Satellite remote sensing images contain complex and diverse ground object information and the images exhibit spatial multi-scale characteristics, making the panoptic segmentation of satellite remote sensing images a highly challenging task. Due to the lack of large-scale annotated datasets for panor...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-10-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/16/21/4002 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850192957476438016 |
|---|---|
| author | Zhuoran Liu Zizhen Li Ying Liang Claudio Persello Bo Sun Guangjun He Lei Ma |
| author_facet | Zhuoran Liu Zizhen Li Ying Liang Claudio Persello Bo Sun Guangjun He Lei Ma |
| author_sort | Zhuoran Liu |
| collection | DOAJ |
| description | Satellite remote sensing images contain complex and diverse ground object information and the images exhibit spatial multi-scale characteristics, making the panoptic segmentation of satellite remote sensing images a highly challenging task. Due to the lack of large-scale annotated datasets for panoramic segmentation, existing methods still suffer from weak model generalization capabilities. To mitigate this issue, this paper leverages the advantages of the Segment Anything Model (SAM), which can segment any object in remote sensing images without requiring any annotations and proposes a high-resolution remote sensing image panoptic segmentation method called Remote Sensing Panoptic Segmentation SAM (RSPS-SAM). Firstly, to address the problem of global information loss caused by cropping large remote sensing images for training, a Batch Attention Pyramid was designed to extract multi-scale features from remote sensing images and capture long-range contextual information between cropped patches, thereby enhancing the semantic understanding of remote sensing images. Secondly, we constructed a Mask Decoder to address the limitation of SAM requiring manual input prompts and its inability to output category information. This decoder utilized mask-based attention for mask segmentation, enabling automatic prompt generation and category prediction of segmented objects. Finally, the effectiveness of the proposed method was validated on the high-resolution remote sensing image airport scene dataset RSAPS-ASD. The results demonstrate that the proposed method achieves segmentation and recognition of foreground instances and background regions in high-resolution remote sensing images without the need for prompt input, while providing smooth segmentation boundaries with a panoptic segmentation quality (PQ) of 57.2, outperforming current mainstream methods. |
| format | Article |
| id | doaj-art-611237c19eb84915aa63d6054aa2d3c6 |
| institution | OA Journals |
| issn | 2072-4292 |
| language | English |
| publishDate | 2024-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj-art-611237c19eb84915aa63d6054aa2d3c62025-08-20T02:14:23ZengMDPI AGRemote Sensing2072-42922024-10-011621400210.3390/rs16214002RSPS-SAM: A Remote Sensing Image Panoptic Segmentation Method Based on SAMZhuoran Liu0Zizhen Li1Ying Liang2Claudio Persello3Bo Sun4Guangjun He5Lei Ma6School of Geodesy and Geomatics, Beijing University of Civil Engineering and Architecture, Beijing 102627, ChinaState Key Laboratory of Space-Earth Integrated Information Technology, Beijing Institute of Satellite Information Engineering, Beijing 100095, ChinaState Key Laboratory of Space-Earth Integrated Information Technology, Beijing Institute of Satellite Information Engineering, Beijing 100095, ChinaDepartment of Earth Observation Science, Faculty ITC, University of Twente, 7500AE Enschede, The NetherlandsCollege of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, ChinaState Key Laboratory of Space-Earth Integrated Information Technology, Beijing Institute of Satellite Information Engineering, Beijing 100095, ChinaSchool of Geography and Ocean Science, Nanjing University, Nanjing 210023, ChinaSatellite remote sensing images contain complex and diverse ground object information and the images exhibit spatial multi-scale characteristics, making the panoptic segmentation of satellite remote sensing images a highly challenging task. Due to the lack of large-scale annotated datasets for panoramic segmentation, existing methods still suffer from weak model generalization capabilities. To mitigate this issue, this paper leverages the advantages of the Segment Anything Model (SAM), which can segment any object in remote sensing images without requiring any annotations and proposes a high-resolution remote sensing image panoptic segmentation method called Remote Sensing Panoptic Segmentation SAM (RSPS-SAM). Firstly, to address the problem of global information loss caused by cropping large remote sensing images for training, a Batch Attention Pyramid was designed to extract multi-scale features from remote sensing images and capture long-range contextual information between cropped patches, thereby enhancing the semantic understanding of remote sensing images. Secondly, we constructed a Mask Decoder to address the limitation of SAM requiring manual input prompts and its inability to output category information. This decoder utilized mask-based attention for mask segmentation, enabling automatic prompt generation and category prediction of segmented objects. Finally, the effectiveness of the proposed method was validated on the high-resolution remote sensing image airport scene dataset RSAPS-ASD. The results demonstrate that the proposed method achieves segmentation and recognition of foreground instances and background regions in high-resolution remote sensing images without the need for prompt input, while providing smooth segmentation boundaries with a panoptic segmentation quality (PQ) of 57.2, outperforming current mainstream methods.https://www.mdpi.com/2072-4292/16/21/4002panoptic segmentationsegment anything modelremote sensingdeep learning |
| spellingShingle | Zhuoran Liu Zizhen Li Ying Liang Claudio Persello Bo Sun Guangjun He Lei Ma RSPS-SAM: A Remote Sensing Image Panoptic Segmentation Method Based on SAM Remote Sensing panoptic segmentation segment anything model remote sensing deep learning |
| title | RSPS-SAM: A Remote Sensing Image Panoptic Segmentation Method Based on SAM |
| title_full | RSPS-SAM: A Remote Sensing Image Panoptic Segmentation Method Based on SAM |
| title_fullStr | RSPS-SAM: A Remote Sensing Image Panoptic Segmentation Method Based on SAM |
| title_full_unstemmed | RSPS-SAM: A Remote Sensing Image Panoptic Segmentation Method Based on SAM |
| title_short | RSPS-SAM: A Remote Sensing Image Panoptic Segmentation Method Based on SAM |
| title_sort | rsps sam a remote sensing image panoptic segmentation method based on sam |
| topic | panoptic segmentation segment anything model remote sensing deep learning |
| url | https://www.mdpi.com/2072-4292/16/21/4002 |
| work_keys_str_mv | AT zhuoranliu rspssamaremotesensingimagepanopticsegmentationmethodbasedonsam AT zizhenli rspssamaremotesensingimagepanopticsegmentationmethodbasedonsam AT yingliang rspssamaremotesensingimagepanopticsegmentationmethodbasedonsam AT claudiopersello rspssamaremotesensingimagepanopticsegmentationmethodbasedonsam AT bosun rspssamaremotesensingimagepanopticsegmentationmethodbasedonsam AT guangjunhe rspssamaremotesensingimagepanopticsegmentationmethodbasedonsam AT leima rspssamaremotesensingimagepanopticsegmentationmethodbasedonsam |