SAM-Net: Spatio-Temporal Sequence Typhoon Cloud Image Prediction Net with Self-Attention Memory
Cloud image prediction is a spatio-temporal sequence prediction task, similar to video prediction. Spatio-temporal sequence prediction involves learning from historical data and using the learned features to generate future images. In this process, the changes in time and space are crucial for spati...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-11-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/16/22/4213 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850147191330439168 |
|---|---|
| author | Yanzhao Ren Jinyuan Ye Xiaochuan Wang Fengjin Xiao Ruijun Liu |
| author_facet | Yanzhao Ren Jinyuan Ye Xiaochuan Wang Fengjin Xiao Ruijun Liu |
| author_sort | Yanzhao Ren |
| collection | DOAJ |
| description | Cloud image prediction is a spatio-temporal sequence prediction task, similar to video prediction. Spatio-temporal sequence prediction involves learning from historical data and using the learned features to generate future images. In this process, the changes in time and space are crucial for spatio-temporal sequence prediction models. However, most models now rely on stacking convolutional layers to obtain local spatial features. In response to the complex changes in cloud position and shape in cloud images, the prediction module of the model needs to be able to extract both global and local spatial features from the cloud images. In addition, for irregular cloud motion, more attention should be paid to the spatio-temporal sequence features between input cloud image frames in the temporal sequence prediction module, considering the extraction of temporal features with long temporal dependencies, so that the spatio-temporal sequence prediction network can learn cloud motion trends more accurately. To address these issues, we have introduced an innovative model called SAM-Net. The self-attention module of this model aims to extract an inner image frame’s spatial features of global and local dependencies. In addition, a memory mechanism has been added to the self-attention module to extract interframe features with long temporal and spatial dependencies. Our method shows better performance than the PredRNN-v2 model on publicly available datasets such as MovingMNIST and KTH. We achieved the best performance in both the 4-time-step and 10-time-step typhoon cloud image predictions. On a cloud dataset consisting of 10 time steps, we observed a decrease in MSE of 180.58, a decrease in LPIPS of 0.064, an increase in SSIM of 0.351, and a significant improvement in PSNR of 5.56 compared to PredRNN-v2. |
| format | Article |
| id | doaj-art-ae4e12f8ba104f08a892fae6504e6f57 |
| institution | OA Journals |
| issn | 2072-4292 |
| language | English |
| publishDate | 2024-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj-art-ae4e12f8ba104f08a892fae6504e6f572025-08-20T02:27:38ZengMDPI AGRemote Sensing2072-42922024-11-011622421310.3390/rs16224213SAM-Net: Spatio-Temporal Sequence Typhoon Cloud Image Prediction Net with Self-Attention MemoryYanzhao Ren0Jinyuan Ye1Xiaochuan Wang2Fengjin Xiao3Ruijun Liu4School of Computer and Artificial Intelligence, Beijing Technology and Business University, Beijing 100048, ChinaSchool of Computer and Artificial Intelligence, Beijing Technology and Business University, Beijing 100048, ChinaSchool of Computer and Artificial Intelligence, Beijing Technology and Business University, Beijing 100048, ChinaChina Meteorological Administration, National Climate Center, Beijing 100081, ChinaSchool of Software, Beihang University, Beijing 100191, ChinaCloud image prediction is a spatio-temporal sequence prediction task, similar to video prediction. Spatio-temporal sequence prediction involves learning from historical data and using the learned features to generate future images. In this process, the changes in time and space are crucial for spatio-temporal sequence prediction models. However, most models now rely on stacking convolutional layers to obtain local spatial features. In response to the complex changes in cloud position and shape in cloud images, the prediction module of the model needs to be able to extract both global and local spatial features from the cloud images. In addition, for irregular cloud motion, more attention should be paid to the spatio-temporal sequence features between input cloud image frames in the temporal sequence prediction module, considering the extraction of temporal features with long temporal dependencies, so that the spatio-temporal sequence prediction network can learn cloud motion trends more accurately. To address these issues, we have introduced an innovative model called SAM-Net. The self-attention module of this model aims to extract an inner image frame’s spatial features of global and local dependencies. In addition, a memory mechanism has been added to the self-attention module to extract interframe features with long temporal and spatial dependencies. Our method shows better performance than the PredRNN-v2 model on publicly available datasets such as MovingMNIST and KTH. We achieved the best performance in both the 4-time-step and 10-time-step typhoon cloud image predictions. On a cloud dataset consisting of 10 time steps, we observed a decrease in MSE of 180.58, a decrease in LPIPS of 0.064, an increase in SSIM of 0.351, and a significant improvement in PSNR of 5.56 compared to PredRNN-v2.https://www.mdpi.com/2072-4292/16/22/4213spatio-temporal sequence predictive learningremote sensingrecurrent neural networksself-attention memory |
| spellingShingle | Yanzhao Ren Jinyuan Ye Xiaochuan Wang Fengjin Xiao Ruijun Liu SAM-Net: Spatio-Temporal Sequence Typhoon Cloud Image Prediction Net with Self-Attention Memory Remote Sensing spatio-temporal sequence predictive learning remote sensing recurrent neural networks self-attention memory |
| title | SAM-Net: Spatio-Temporal Sequence Typhoon Cloud Image Prediction Net with Self-Attention Memory |
| title_full | SAM-Net: Spatio-Temporal Sequence Typhoon Cloud Image Prediction Net with Self-Attention Memory |
| title_fullStr | SAM-Net: Spatio-Temporal Sequence Typhoon Cloud Image Prediction Net with Self-Attention Memory |
| title_full_unstemmed | SAM-Net: Spatio-Temporal Sequence Typhoon Cloud Image Prediction Net with Self-Attention Memory |
| title_short | SAM-Net: Spatio-Temporal Sequence Typhoon Cloud Image Prediction Net with Self-Attention Memory |
| title_sort | sam net spatio temporal sequence typhoon cloud image prediction net with self attention memory |
| topic | spatio-temporal sequence predictive learning remote sensing recurrent neural networks self-attention memory |
| url | https://www.mdpi.com/2072-4292/16/22/4213 |
| work_keys_str_mv | AT yanzhaoren samnetspatiotemporalsequencetyphooncloudimagepredictionnetwithselfattentionmemory AT jinyuanye samnetspatiotemporalsequencetyphooncloudimagepredictionnetwithselfattentionmemory AT xiaochuanwang samnetspatiotemporalsequencetyphooncloudimagepredictionnetwithselfattentionmemory AT fengjinxiao samnetspatiotemporalsequencetyphooncloudimagepredictionnetwithselfattentionmemory AT ruijunliu samnetspatiotemporalsequencetyphooncloudimagepredictionnetwithselfattentionmemory |