Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responses
Abstract Chatbot-based multimodal AI holds promise for collecting medical histories and diagnosing ophthalmic diseases using textual and imaging data. This study developed and evaluated the ChatGPT-powered Intelligent Ophthalmic Multimodal Interactive Diagnostic System (IOMIDS) to enable patient sel...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-01-01
|
| Series: | npj Digital Medicine |
| Online Access: | https://doi.org/10.1038/s41746-025-01461-0 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832571242472800256 |
|---|---|
| author | Ruiqi Ma Qian Cheng Jing Yao Zhiyu Peng Mingxu Yan Jie Lu Jingjing Liao Lejin Tian Wenjun Shu Yunqiu Zhang Jinghan Wang Pengfei Jiang Weiyi Xia Xiaofeng Li Lu Gan Yue Zhao Jiang Zhu Bing Qin Qin Jiang Xiawei Wang Xintong Lin Haifeng Chen Weifang Zhu Dehui Xiang Baoqing Nie Jingtao Wang Jie Guo Kang Xue Hongguang Cui Jinwei Cheng Xiangjia Zhu Jiaxu Hong Fei Shi Rui Zhang Xinjian Chen Chen Zhao |
| author_facet | Ruiqi Ma Qian Cheng Jing Yao Zhiyu Peng Mingxu Yan Jie Lu Jingjing Liao Lejin Tian Wenjun Shu Yunqiu Zhang Jinghan Wang Pengfei Jiang Weiyi Xia Xiaofeng Li Lu Gan Yue Zhao Jiang Zhu Bing Qin Qin Jiang Xiawei Wang Xintong Lin Haifeng Chen Weifang Zhu Dehui Xiang Baoqing Nie Jingtao Wang Jie Guo Kang Xue Hongguang Cui Jinwei Cheng Xiangjia Zhu Jiaxu Hong Fei Shi Rui Zhang Xinjian Chen Chen Zhao |
| author_sort | Ruiqi Ma |
| collection | DOAJ |
| description | Abstract Chatbot-based multimodal AI holds promise for collecting medical histories and diagnosing ophthalmic diseases using textual and imaging data. This study developed and evaluated the ChatGPT-powered Intelligent Ophthalmic Multimodal Interactive Diagnostic System (IOMIDS) to enable patient self-diagnosis and self-triage. IOMIDS included a text model and three multimodal models (text + slit-lamp, text + smartphone, text + slit-lamp + smartphone). The performance was evaluated through a two-stage cross-sectional study across three medical centers involving 10 subspecialties and 50 diseases. Using 15640 data entries, IOMIDS actively collected and analyzed medical history alongside slit-lamp and/or smartphone images. The text + smartphone model showed the highest diagnostic accuracy (internal: 79.6%, external: 81.1%), while other multimodal models underperformed or matched the text model (internal: 69.6%, external: 72.5%). Moreover, triage accuracy was consistent across models. Multimodal approaches enhanced response quality and reduced misinformation. This proof-of-concept study highlights the potential of chatbot-based multimodal AI for self-diagnosis and self-triage. (The clinical trial was registered on June 26, 2023, on ClinicalTrials.gov under the registration number NCT05930444.). |
| format | Article |
| id | doaj-art-71de791e026848b48177c9ed49c87e83 |
| institution | Kabale University |
| issn | 2398-6352 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | npj Digital Medicine |
| spelling | doaj-art-71de791e026848b48177c9ed49c87e832025-02-02T12:43:46ZengNature Portfolionpj Digital Medicine2398-63522025-01-018111810.1038/s41746-025-01461-0Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responsesRuiqi Ma0Qian Cheng1Jing Yao2Zhiyu Peng3Mingxu Yan4Jie Lu5Jingjing Liao6Lejin Tian7Wenjun Shu8Yunqiu Zhang9Jinghan Wang10Pengfei Jiang11Weiyi Xia12Xiaofeng Li13Lu Gan14Yue Zhao15Jiang Zhu16Bing Qin17Qin Jiang18Xiawei Wang19Xintong Lin20Haifeng Chen21Weifang Zhu22Dehui Xiang23Baoqing Nie24Jingtao Wang25Jie Guo26Kang Xue27Hongguang Cui28Jinwei Cheng29Xiangjia Zhu30Jiaxu Hong31Fei Shi32Rui Zhang33Xinjian Chen34Chen Zhao35Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityMIPAV Lab, School of Electronics and Information Engineering, Soochow UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityMIPAV Lab, School of Electronics and Information Engineering, Soochow UniversityState Key Laboratory of Genetic Engineering, Department of Computational Biology, School of Life Sciences, Fudan UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityDepartment of Epidemiology, School of Public Health, and The Key Laboratory of Public Health Safety of Ministry of Education, Fudan UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityShanghai Jiao Tong University Instrument Analysis CenterEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityThe Affiliated Eye Hospital, Nanjing Medical UniversityDepartment of Ophthalmology, Suqian First HospitalDepartment of Ophthalmology, Suqian First HospitalThe Affiliated Eye Hospital, Nanjing Medical UniversityDepartment of Ophthalmology, The First Affiliated Hospital, Zhejiang University School of MedicineEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityMIPAV Lab, School of Electronics and Information Engineering, Soochow UniversityMIPAV Lab, School of Electronics and Information Engineering, Soochow UniversityMIPAV Lab, School of Electronics and Information Engineering, Soochow UniversityMIPAV Lab, School of Electronics and Information Engineering, Soochow UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityDepartment of Ophthalmology, The First Affiliated Hospital, Zhejiang University School of MedicineEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityMIPAV Lab, School of Electronics and Information Engineering, Soochow UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityMIPAV Lab, School of Electronics and Information Engineering, Soochow UniversityEye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan UniversityAbstract Chatbot-based multimodal AI holds promise for collecting medical histories and diagnosing ophthalmic diseases using textual and imaging data. This study developed and evaluated the ChatGPT-powered Intelligent Ophthalmic Multimodal Interactive Diagnostic System (IOMIDS) to enable patient self-diagnosis and self-triage. IOMIDS included a text model and three multimodal models (text + slit-lamp, text + smartphone, text + slit-lamp + smartphone). The performance was evaluated through a two-stage cross-sectional study across three medical centers involving 10 subspecialties and 50 diseases. Using 15640 data entries, IOMIDS actively collected and analyzed medical history alongside slit-lamp and/or smartphone images. The text + smartphone model showed the highest diagnostic accuracy (internal: 79.6%, external: 81.1%), while other multimodal models underperformed or matched the text model (internal: 69.6%, external: 72.5%). Moreover, triage accuracy was consistent across models. Multimodal approaches enhanced response quality and reduced misinformation. This proof-of-concept study highlights the potential of chatbot-based multimodal AI for self-diagnosis and self-triage. (The clinical trial was registered on June 26, 2023, on ClinicalTrials.gov under the registration number NCT05930444.).https://doi.org/10.1038/s41746-025-01461-0 |
| spellingShingle | Ruiqi Ma Qian Cheng Jing Yao Zhiyu Peng Mingxu Yan Jie Lu Jingjing Liao Lejin Tian Wenjun Shu Yunqiu Zhang Jinghan Wang Pengfei Jiang Weiyi Xia Xiaofeng Li Lu Gan Yue Zhao Jiang Zhu Bing Qin Qin Jiang Xiawei Wang Xintong Lin Haifeng Chen Weifang Zhu Dehui Xiang Baoqing Nie Jingtao Wang Jie Guo Kang Xue Hongguang Cui Jinwei Cheng Xiangjia Zhu Jiaxu Hong Fei Shi Rui Zhang Xinjian Chen Chen Zhao Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responses npj Digital Medicine |
| title | Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responses |
| title_full | Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responses |
| title_fullStr | Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responses |
| title_full_unstemmed | Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responses |
| title_short | Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responses |
| title_sort | multimodal machine learning enables ai chatbot to diagnose ophthalmic diseases and provide high quality medical responses |
| url | https://doi.org/10.1038/s41746-025-01461-0 |
| work_keys_str_mv | AT ruiqima multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT qiancheng multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT jingyao multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT zhiyupeng multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT mingxuyan multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT jielu multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT jingjingliao multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT lejintian multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT wenjunshu multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT yunqiuzhang multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT jinghanwang multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT pengfeijiang multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT weiyixia multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT xiaofengli multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT lugan multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT yuezhao multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT jiangzhu multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT bingqin multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT qinjiang multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT xiaweiwang multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT xintonglin multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT haifengchen multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT weifangzhu multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT dehuixiang multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT baoqingnie multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT jingtaowang multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT jieguo multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT kangxue multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT hongguangcui multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT jinweicheng multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT xiangjiazhu multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT jiaxuhong multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT feishi multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT ruizhang multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT xinjianchen multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses AT chenzhao multimodalmachinelearningenablesaichatbottodiagnoseophthalmicdiseasesandprovidehighqualitymedicalresponses |