Hand-aware graph convolution network for skeleton-based sign language recognition
Skeleton-based sign language recognition (SLR) is a challenging research area mainly due to the fast and complex hand movement. Currently, graph convolution networks (GCNs) have been employed in skeleton-based SLR and achieved remarkable performance. However, existing GCN-based SLR methods suffer fr...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
KeAi Communications Co., Ltd.
2025-01-01
|
| Series: | Journal of Information and Intelligence |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S294971592400074X |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849253334647570432 |
|---|---|
| author | Juan Song Huixuechun Wang Jianan Li Jian Zheng Zhifu Zhao Qingshan Li |
| author_facet | Juan Song Huixuechun Wang Jianan Li Jian Zheng Zhifu Zhao Qingshan Li |
| author_sort | Juan Song |
| collection | DOAJ |
| description | Skeleton-based sign language recognition (SLR) is a challenging research area mainly due to the fast and complex hand movement. Currently, graph convolution networks (GCNs) have been employed in skeleton-based SLR and achieved remarkable performance. However, existing GCN-based SLR methods suffer from a lack of explicit attention to hand topology which plays an important role in the sign language representation. To address this issue, we propose a novel hand-aware graph convolution network (HA-GCN) to focus on hand topological relationships of skeleton graph. Specifically, a hand-aware graph convolution layer is designed to capture both global body and local hand information, in which two sub-graphs are defined and incorporated to represent hand topology information. In addition, in order to eliminate the over-fitting problem, an adaptive DropGraph is designed in construction of hand-aware graph convolution block to remove the spatial and temporal redundancy in the sign language representation. With the aim to further improve the performance, the joints information, bones, together with their motion information are simultaneously modeled in a multi-stream framework. Extensive experiments on the two open-source datasets, AUTSL and INCLUDE, demonstrate that our proposed algorithm outperforms the state-of-the-art with a significant margin. Our code is available at https://github.com/snorlaxse/HA-SLR-GCN. |
| format | Article |
| id | doaj-art-6b2fab6acb77458e816efeaa5314071a |
| institution | Kabale University |
| issn | 2949-7159 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | KeAi Communications Co., Ltd. |
| record_format | Article |
| series | Journal of Information and Intelligence |
| spelling | doaj-art-6b2fab6acb77458e816efeaa5314071a2025-08-20T03:56:23ZengKeAi Communications Co., Ltd.Journal of Information and Intelligence2949-71592025-01-0131365010.1016/j.jiixd.2024.08.001Hand-aware graph convolution network for skeleton-based sign language recognitionJuan Song0Huixuechun Wang1Jianan Li2Jian Zheng3Zhifu Zhao4Qingshan Li5School of Computer Science and Technology, Xidian University, Xi'an 710126, ChinaSchool of Computer Science and Technology, Xidian University, Xi'an 710126, ChinaSchool of Computer Science and Technology, Xidian University, Xi'an 710126, China; Corresponding author.School of Computer Science and Technology, Xidian University, Xi'an 710126, ChinaSchool of Artificial Intelligence Engineering, Xidian University, Xi'an 710126, ChinaSchool of Computer Science and Technology, Xidian University, Xi'an 710126, ChinaSkeleton-based sign language recognition (SLR) is a challenging research area mainly due to the fast and complex hand movement. Currently, graph convolution networks (GCNs) have been employed in skeleton-based SLR and achieved remarkable performance. However, existing GCN-based SLR methods suffer from a lack of explicit attention to hand topology which plays an important role in the sign language representation. To address this issue, we propose a novel hand-aware graph convolution network (HA-GCN) to focus on hand topological relationships of skeleton graph. Specifically, a hand-aware graph convolution layer is designed to capture both global body and local hand information, in which two sub-graphs are defined and incorporated to represent hand topology information. In addition, in order to eliminate the over-fitting problem, an adaptive DropGraph is designed in construction of hand-aware graph convolution block to remove the spatial and temporal redundancy in the sign language representation. With the aim to further improve the performance, the joints information, bones, together with their motion information are simultaneously modeled in a multi-stream framework. Extensive experiments on the two open-source datasets, AUTSL and INCLUDE, demonstrate that our proposed algorithm outperforms the state-of-the-art with a significant margin. Our code is available at https://github.com/snorlaxse/HA-SLR-GCN.http://www.sciencedirect.com/science/article/pii/S294971592400074XSign language recognitionGraph convolutional networkHand-aware graphsSkeleton dataMulti-stream fusion |
| spellingShingle | Juan Song Huixuechun Wang Jianan Li Jian Zheng Zhifu Zhao Qingshan Li Hand-aware graph convolution network for skeleton-based sign language recognition Journal of Information and Intelligence Sign language recognition Graph convolutional network Hand-aware graphs Skeleton data Multi-stream fusion |
| title | Hand-aware graph convolution network for skeleton-based sign language recognition |
| title_full | Hand-aware graph convolution network for skeleton-based sign language recognition |
| title_fullStr | Hand-aware graph convolution network for skeleton-based sign language recognition |
| title_full_unstemmed | Hand-aware graph convolution network for skeleton-based sign language recognition |
| title_short | Hand-aware graph convolution network for skeleton-based sign language recognition |
| title_sort | hand aware graph convolution network for skeleton based sign language recognition |
| topic | Sign language recognition Graph convolutional network Hand-aware graphs Skeleton data Multi-stream fusion |
| url | http://www.sciencedirect.com/science/article/pii/S294971592400074X |
| work_keys_str_mv | AT juansong handawaregraphconvolutionnetworkforskeletonbasedsignlanguagerecognition AT huixuechunwang handawaregraphconvolutionnetworkforskeletonbasedsignlanguagerecognition AT jiananli handawaregraphconvolutionnetworkforskeletonbasedsignlanguagerecognition AT jianzheng handawaregraphconvolutionnetworkforskeletonbasedsignlanguagerecognition AT zhifuzhao handawaregraphconvolutionnetworkforskeletonbasedsignlanguagerecognition AT qingshanli handawaregraphconvolutionnetworkforskeletonbasedsignlanguagerecognition |