ECG-LM: Understanding Electrocardiogram with a Large Language Model
Background: The electrocardiogram (ECG) is a valuable, noninvasive tool for monitoring heart-related conditions, providing critical insights. However, the interpretation of ECG data alongside patient information demands substantial medical expertise and resources. While deep learning methods help st...
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
American Association for the Advancement of Science (AAAS)
2025-01-01
|
| Series: | Health Data Science |
| Online Access: | https://spj.science.org/doi/10.34133/hds.0221 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832541091554918400 |
|---|---|
| author | Kai Yang Massimo Hong Jiahuan Zhang Yizhen Luo Suyuan Zhao Ou Zhang Xiaomao Yu Jiawen Zhou Liuqing Yang Ping Zhang Mu Qiao Zaiqing Nie |
| author_facet | Kai Yang Massimo Hong Jiahuan Zhang Yizhen Luo Suyuan Zhao Ou Zhang Xiaomao Yu Jiawen Zhou Liuqing Yang Ping Zhang Mu Qiao Zaiqing Nie |
| author_sort | Kai Yang |
| collection | DOAJ |
| description | Background: The electrocardiogram (ECG) is a valuable, noninvasive tool for monitoring heart-related conditions, providing critical insights. However, the interpretation of ECG data alongside patient information demands substantial medical expertise and resources. While deep learning methods help streamline this process, they often fall short in integrating patient data with ECG readings and do not provide the nuanced clinical suggestions and insights necessary for accurate diagnosis. Methods: Although recent advancements in multi-modal large language modeling have propelled their application scope beyond the natural language processing domain, their applicability to ECG processing remains largely unexplored, partly due to the lack of text–ECG data. To this end, we develop ECG-Language Model (ECG-LM), the first multi-modal large language model able to process natural language and understand ECG signals. The model employs a specialized ECG encoder that transforms raw ECG signals into a high-dimensional feature space, which is then aligned with the textual feature space derived from the large language model. To address the scarcity of text–ECG data, we generated text–ECG pairs by leveraging detailed ECG pattern descriptions from medical guidelines, creating a robust dataset for pre-training ECG-LM. Additionally, we fine-tune ECG-LM with public clinical conversation datasets and build an additional supervised fine-tuning dataset based on real clinical data from the hospital, aiming to provide a more comprehensive and customized user experience. Results: ECG-LM outperforms existing few-shot and zero-shot solutions in cardiovascular disease detection across all 3 tasks (diagnostic, rhythm, and form) while also demonstrating strong potential in ECG-related question answering. Conclusions: The results across various tasks demonstrate that ECG-LM effectively captures the intricate features of ECGs, showcasing its versatility in applications such as disease prediction and advanced question answering. |
| format | Article |
| id | doaj-art-af241abe9a6248fb8b2088869f683705 |
| institution | Kabale University |
| issn | 2765-8783 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | American Association for the Advancement of Science (AAAS) |
| record_format | Article |
| series | Health Data Science |
| spelling | doaj-art-af241abe9a6248fb8b2088869f6837052025-02-04T09:53:00ZengAmerican Association for the Advancement of Science (AAAS)Health Data Science2765-87832025-01-01510.34133/hds.0221ECG-LM: Understanding Electrocardiogram with a Large Language ModelKai Yang0Massimo Hong1Jiahuan Zhang2Yizhen Luo3Suyuan Zhao4Ou Zhang5Xiaomao Yu6Jiawen Zhou7Liuqing Yang8Ping Zhang9Mu Qiao10Zaiqing Nie11Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China.Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China.Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China.Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China.Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China.Beijing Tsinghua Changgung Hospital, Beijing, China.Beijing Tsinghua Changgung Hospital, Beijing, China.Beijing Tsinghua Changgung Hospital, Beijing, China.Beijing Tsinghua Changgung Hospital, Beijing, China.Beijing Tsinghua Changgung Hospital, Beijing, China.PharMolix Inc., Beijing, China.Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China.Background: The electrocardiogram (ECG) is a valuable, noninvasive tool for monitoring heart-related conditions, providing critical insights. However, the interpretation of ECG data alongside patient information demands substantial medical expertise and resources. While deep learning methods help streamline this process, they often fall short in integrating patient data with ECG readings and do not provide the nuanced clinical suggestions and insights necessary for accurate diagnosis. Methods: Although recent advancements in multi-modal large language modeling have propelled their application scope beyond the natural language processing domain, their applicability to ECG processing remains largely unexplored, partly due to the lack of text–ECG data. To this end, we develop ECG-Language Model (ECG-LM), the first multi-modal large language model able to process natural language and understand ECG signals. The model employs a specialized ECG encoder that transforms raw ECG signals into a high-dimensional feature space, which is then aligned with the textual feature space derived from the large language model. To address the scarcity of text–ECG data, we generated text–ECG pairs by leveraging detailed ECG pattern descriptions from medical guidelines, creating a robust dataset for pre-training ECG-LM. Additionally, we fine-tune ECG-LM with public clinical conversation datasets and build an additional supervised fine-tuning dataset based on real clinical data from the hospital, aiming to provide a more comprehensive and customized user experience. Results: ECG-LM outperforms existing few-shot and zero-shot solutions in cardiovascular disease detection across all 3 tasks (diagnostic, rhythm, and form) while also demonstrating strong potential in ECG-related question answering. Conclusions: The results across various tasks demonstrate that ECG-LM effectively captures the intricate features of ECGs, showcasing its versatility in applications such as disease prediction and advanced question answering.https://spj.science.org/doi/10.34133/hds.0221 |
| spellingShingle | Kai Yang Massimo Hong Jiahuan Zhang Yizhen Luo Suyuan Zhao Ou Zhang Xiaomao Yu Jiawen Zhou Liuqing Yang Ping Zhang Mu Qiao Zaiqing Nie ECG-LM: Understanding Electrocardiogram with a Large Language Model Health Data Science |
| title | ECG-LM: Understanding Electrocardiogram with a Large Language Model |
| title_full | ECG-LM: Understanding Electrocardiogram with a Large Language Model |
| title_fullStr | ECG-LM: Understanding Electrocardiogram with a Large Language Model |
| title_full_unstemmed | ECG-LM: Understanding Electrocardiogram with a Large Language Model |
| title_short | ECG-LM: Understanding Electrocardiogram with a Large Language Model |
| title_sort | ecg lm understanding electrocardiogram with a large language model |
| url | https://spj.science.org/doi/10.34133/hds.0221 |
| work_keys_str_mv | AT kaiyang ecglmunderstandingelectrocardiogramwithalargelanguagemodel AT massimohong ecglmunderstandingelectrocardiogramwithalargelanguagemodel AT jiahuanzhang ecglmunderstandingelectrocardiogramwithalargelanguagemodel AT yizhenluo ecglmunderstandingelectrocardiogramwithalargelanguagemodel AT suyuanzhao ecglmunderstandingelectrocardiogramwithalargelanguagemodel AT ouzhang ecglmunderstandingelectrocardiogramwithalargelanguagemodel AT xiaomaoyu ecglmunderstandingelectrocardiogramwithalargelanguagemodel AT jiawenzhou ecglmunderstandingelectrocardiogramwithalargelanguagemodel AT liuqingyang ecglmunderstandingelectrocardiogramwithalargelanguagemodel AT pingzhang ecglmunderstandingelectrocardiogramwithalargelanguagemodel AT muqiao ecglmunderstandingelectrocardiogramwithalargelanguagemodel AT zaiqingnie ecglmunderstandingelectrocardiogramwithalargelanguagemodel |