Semantic Tokenization-Based Mamba for Hyperspectral Image Classification
Recently, the transformer-based model has shown superior performance in hyperspectral image classification (HSIC) due to its excellent ability to model long-term dependencies on sequence data. An important component of the transformer is the tokenizer, which can transform the features into semantic...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10838328/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832576756294352896 |
|---|---|
| author | Ri Ming Na Chen Jiangtao Peng Weiwei Sun Zhijing Ye |
| author_facet | Ri Ming Na Chen Jiangtao Peng Weiwei Sun Zhijing Ye |
| author_sort | Ri Ming |
| collection | DOAJ |
| description | Recently, the transformer-based model has shown superior performance in hyperspectral image classification (HSIC) due to its excellent ability to model long-term dependencies on sequence data. An important component of the transformer is the tokenizer, which can transform the features into semantic token sequences (STS). Nonetheless, transformer's semantic tokenization strategy is hardly representative of local relatively important high-level semantics because of its global receptive field. Recently, the Mamba-based methods have shown even stronger spatial context modeling ability than Transformer for HSIC. However, these Mamba-based methods mainly focus on spectral and spatial dimensions. They tend to extract semantic information in very long feature sequences or represent semantic information in several typical tokens, which may ignore some important semantics of the HSIs. In order to represent the semantic information of HSIs more holistically in Mamba, this article proposes a semantic tokenization-based Mamba (STMamba) model. In STMamba, a spectral-spatial feature extraction module is used to extract the spectral–spatial joint features. Then, a generated semantic token sequences module is designed to transform the features into STS. Subsequently, the STS are fed into the semantic token state spatial model to capture relationships between different semantic tokens. Finally, the fused semantic token is passed into a classifier for classification. Experimental results on three HSI datasets demonstrate that the proposed STMamba outperforms existing state-of-the-art deep learning and transformer-based methods. |
| format | Article |
| id | doaj-art-5aabbb64754944588b2c65141d0e8652 |
| institution | Kabale University |
| issn | 1939-1404 2151-1535 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj-art-5aabbb64754944588b2c65141d0e86522025-01-31T00:00:19ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing1939-14042151-15352025-01-01184227424110.1109/JSTARS.2025.352812210838328Semantic Tokenization-Based Mamba for Hyperspectral Image ClassificationRi Ming0https://orcid.org/0009-0006-5375-3825Na Chen1https://orcid.org/0009-0003-4672-3247Jiangtao Peng2https://orcid.org/0000-0002-4759-0584Weiwei Sun3https://orcid.org/0000-0003-3399-7858Zhijing Ye4https://orcid.org/0000-0002-0369-4795Hubei Key Laboratory of Applied Mathematics, Faculty of Mathematics and Statistics, the Key Laboratory of Intelligent Sensing System and Security, Ministry of Education, Hubei University, Wuhan, ChinaHubei Key Laboratory of Applied Mathematics, Faculty of Mathematics and Statistics, the Key Laboratory of Intelligent Sensing System and Security, Ministry of Education, Hubei University, Wuhan, ChinaHubei Key Laboratory of Applied Mathematics, Faculty of Mathematics and Statistics, the Key Laboratory of Intelligent Sensing System and Security, Ministry of Education, Hubei University, Wuhan, ChinaDepartment of Geography and Spatial Information Techniques, Ningbo University, Ningbo, ChinaFaculty of Innovation Engineering, Macau University of Science and Technology, Taipa, MacauRecently, the transformer-based model has shown superior performance in hyperspectral image classification (HSIC) due to its excellent ability to model long-term dependencies on sequence data. An important component of the transformer is the tokenizer, which can transform the features into semantic token sequences (STS). Nonetheless, transformer's semantic tokenization strategy is hardly representative of local relatively important high-level semantics because of its global receptive field. Recently, the Mamba-based methods have shown even stronger spatial context modeling ability than Transformer for HSIC. However, these Mamba-based methods mainly focus on spectral and spatial dimensions. They tend to extract semantic information in very long feature sequences or represent semantic information in several typical tokens, which may ignore some important semantics of the HSIs. In order to represent the semantic information of HSIs more holistically in Mamba, this article proposes a semantic tokenization-based Mamba (STMamba) model. In STMamba, a spectral-spatial feature extraction module is used to extract the spectral–spatial joint features. Then, a generated semantic token sequences module is designed to transform the features into STS. Subsequently, the STS are fed into the semantic token state spatial model to capture relationships between different semantic tokens. Finally, the fused semantic token is passed into a classifier for classification. Experimental results on three HSI datasets demonstrate that the proposed STMamba outperforms existing state-of-the-art deep learning and transformer-based methods.https://ieeexplore.ieee.org/document/10838328/Convolutional neural networkshyperspectral image classificationsemantic token state spatial modeltransformer |
| spellingShingle | Ri Ming Na Chen Jiangtao Peng Weiwei Sun Zhijing Ye Semantic Tokenization-Based Mamba for Hyperspectral Image Classification IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Convolutional neural networks hyperspectral image classification semantic token state spatial model transformer |
| title | Semantic Tokenization-Based Mamba for Hyperspectral Image Classification |
| title_full | Semantic Tokenization-Based Mamba for Hyperspectral Image Classification |
| title_fullStr | Semantic Tokenization-Based Mamba for Hyperspectral Image Classification |
| title_full_unstemmed | Semantic Tokenization-Based Mamba for Hyperspectral Image Classification |
| title_short | Semantic Tokenization-Based Mamba for Hyperspectral Image Classification |
| title_sort | semantic tokenization based mamba for hyperspectral image classification |
| topic | Convolutional neural networks hyperspectral image classification semantic token state spatial model transformer |
| url | https://ieeexplore.ieee.org/document/10838328/ |
| work_keys_str_mv | AT riming semantictokenizationbasedmambaforhyperspectralimageclassification AT nachen semantictokenizationbasedmambaforhyperspectralimageclassification AT jiangtaopeng semantictokenizationbasedmambaforhyperspectralimageclassification AT weiweisun semantictokenizationbasedmambaforhyperspectralimageclassification AT zhijingye semantictokenizationbasedmambaforhyperspectralimageclassification |