Acoustic Inspired Brain-to-Sentence Decoder for Logosyllabic Language
Recent advances in brain–computer interfaces (BCIs) have demonstrated the potential to decode language from brain activity into sound or text, which has predominantly focused on alphabetic languages, such as English. However, logosyllabic languages, such as Mandarin Chinese, present marked challenge...
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
American Association for the Advancement of Science (AAAS)
2025-01-01
|
| Series: | Cyborg and Bionic Systems |
| Online Access: | https://spj.science.org/doi/10.34133/cbsystems.0257 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850140054147563520 |
|---|---|
| author | Chen Feng Lu Cao Di Wu En Zhang Ting Wang Xiaowei Jiang Jinbo Chen Hui Wu Siyu Lin Qiming Hou Junming Zhu Jie Yang Mohamad Sawan Yue Zhang |
| author_facet | Chen Feng Lu Cao Di Wu En Zhang Ting Wang Xiaowei Jiang Jinbo Chen Hui Wu Siyu Lin Qiming Hou Junming Zhu Jie Yang Mohamad Sawan Yue Zhang |
| author_sort | Chen Feng |
| collection | DOAJ |
| description | Recent advances in brain–computer interfaces (BCIs) have demonstrated the potential to decode language from brain activity into sound or text, which has predominantly focused on alphabetic languages, such as English. However, logosyllabic languages, such as Mandarin Chinese, present marked challenges for establishing decoders that cover all characters, due to its unique syllable structures, extended character sets (e.g., over 50,000 characters for Mandarin Chinese), and complex mappings between characters and syllables, thus hindering practical applications. Here, we leverage the acoustic features of Mandarin Chinese syllables, constructing prediction models for syllable components (initials, tones, and finals), and decode speech-related stereoelectroencephalography (sEEG) signals into coherent Chinese sentences. The results demonstrate a high sentence-level offline decoding performance with a median character accuracy of 71.00% over the full spectrum of characters in the best participant. We also verified that incorporating acoustic-related features into the design of prediction models substantially enhances the accuracy of initials, tones, and finals. Moreover, our findings revealed that effective speech decoding also involves subcortical structures like the thalamus in addition to traditional language-related brain regions. Overall, we established a brain-to-sentence decoder for logosyllabic languages over full character set with a large intracranial electroencephalography dataset. |
| format | Article |
| id | doaj-art-350ec8c39e75427db185cea500b9f024 |
| institution | OA Journals |
| issn | 2692-7632 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | American Association for the Advancement of Science (AAAS) |
| record_format | Article |
| series | Cyborg and Bionic Systems |
| spelling | doaj-art-350ec8c39e75427db185cea500b9f0242025-08-20T02:29:59ZengAmerican Association for the Advancement of Science (AAAS)Cyborg and Bionic Systems2692-76322025-01-01610.34133/cbsystems.0257Acoustic Inspired Brain-to-Sentence Decoder for Logosyllabic LanguageChen Feng0Lu Cao1Di Wu2En Zhang3Ting Wang4Xiaowei Jiang5Jinbo Chen6Hui Wu7Siyu Lin8Qiming Hou9Junming Zhu10Jie Yang11Mohamad Sawan12Yue Zhang13Department of Neurosurgery, The Second Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, Zhejiang Province, China.School of Engineering, Westlake University, Hangzhou, Zhejiang Province, China.School of Engineering, Westlake University, Hangzhou, Zhejiang Province, China.State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China.School of Foreign Languages, Tongji University, Shanghai, China.Australian AI Institute, School of Computer Science, Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, Australia.School of Engineering, Westlake University, Hangzhou, Zhejiang Province, China.School of Engineering, Westlake University, Hangzhou, Zhejiang Province, China.School of Engineering, Westlake University, Hangzhou, Zhejiang Province, China.School of Engineering, Westlake University, Hangzhou, Zhejiang Province, China.Department of Neurosurgery, The Second Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, Zhejiang Province, China.School of Engineering, Westlake University, Hangzhou, Zhejiang Province, China.School of Engineering, Westlake University, Hangzhou, Zhejiang Province, China.School of Engineering, Westlake University, Hangzhou, Zhejiang Province, China.Recent advances in brain–computer interfaces (BCIs) have demonstrated the potential to decode language from brain activity into sound or text, which has predominantly focused on alphabetic languages, such as English. However, logosyllabic languages, such as Mandarin Chinese, present marked challenges for establishing decoders that cover all characters, due to its unique syllable structures, extended character sets (e.g., over 50,000 characters for Mandarin Chinese), and complex mappings between characters and syllables, thus hindering practical applications. Here, we leverage the acoustic features of Mandarin Chinese syllables, constructing prediction models for syllable components (initials, tones, and finals), and decode speech-related stereoelectroencephalography (sEEG) signals into coherent Chinese sentences. The results demonstrate a high sentence-level offline decoding performance with a median character accuracy of 71.00% over the full spectrum of characters in the best participant. We also verified that incorporating acoustic-related features into the design of prediction models substantially enhances the accuracy of initials, tones, and finals. Moreover, our findings revealed that effective speech decoding also involves subcortical structures like the thalamus in addition to traditional language-related brain regions. Overall, we established a brain-to-sentence decoder for logosyllabic languages over full character set with a large intracranial electroencephalography dataset.https://spj.science.org/doi/10.34133/cbsystems.0257 |
| spellingShingle | Chen Feng Lu Cao Di Wu En Zhang Ting Wang Xiaowei Jiang Jinbo Chen Hui Wu Siyu Lin Qiming Hou Junming Zhu Jie Yang Mohamad Sawan Yue Zhang Acoustic Inspired Brain-to-Sentence Decoder for Logosyllabic Language Cyborg and Bionic Systems |
| title | Acoustic Inspired Brain-to-Sentence Decoder for Logosyllabic Language |
| title_full | Acoustic Inspired Brain-to-Sentence Decoder for Logosyllabic Language |
| title_fullStr | Acoustic Inspired Brain-to-Sentence Decoder for Logosyllabic Language |
| title_full_unstemmed | Acoustic Inspired Brain-to-Sentence Decoder for Logosyllabic Language |
| title_short | Acoustic Inspired Brain-to-Sentence Decoder for Logosyllabic Language |
| title_sort | acoustic inspired brain to sentence decoder for logosyllabic language |
| url | https://spj.science.org/doi/10.34133/cbsystems.0257 |
| work_keys_str_mv | AT chenfeng acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT lucao acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT diwu acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT enzhang acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT tingwang acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT xiaoweijiang acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT jinbochen acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT huiwu acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT siyulin acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT qiminghou acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT junmingzhu acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT jieyang acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT mohamadsawan acousticinspiredbraintosentencedecoderforlogosyllabiclanguage AT yuezhang acousticinspiredbraintosentencedecoderforlogosyllabiclanguage |