Integrating retrieval-augmented generation for enhanced personalized physician recommendations in web-based medical services: model development study
BackgroundWeb-based medical services have significantly improved access to healthcare by enabling remote consultations, streamlining scheduling, and improving access to medical information. However, providing personalized physician recommendations remains a challenge, often relying on manual triage...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-01-01
|
| Series: | Frontiers in Public Health |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fpubh.2025.1501408/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832582965306064896 |
|---|---|
| author | Yingbin Zheng Yiwei Yan Sai Chen Yunping Cai Kun Ren Yishan Liu Jiaying Zhuang Min Zhao |
| author_facet | Yingbin Zheng Yiwei Yan Sai Chen Yunping Cai Kun Ren Yishan Liu Jiaying Zhuang Min Zhao |
| author_sort | Yingbin Zheng |
| collection | DOAJ |
| description | BackgroundWeb-based medical services have significantly improved access to healthcare by enabling remote consultations, streamlining scheduling, and improving access to medical information. However, providing personalized physician recommendations remains a challenge, often relying on manual triage by schedulers, which can be limited by scalability and availability.ObjectiveThis study aimed to develop and validate a Retrieval-Augmented Generation-Based Physician Recommendation (RAGPR) model for better triage performance.MethodsThis study utilizes a comprehensive dataset consisting of 646,383 consultation records from the Internet Hospital of the First Affiliated Hospital of Xiamen University. The research primarily evaluates the performance of various embedding models, including FastText, SBERT, and OpenAI, for the purposes of clustering and classifying medical condition labels. Additionally, the study assesses the effectiveness of large language models (LLMs) by comparing Mistral, GPT-4o-mini, and GPT-4o. Furthermore, the study includes the participation of three triage staff members who contributed to the evaluation of the efficiency of the RAGPR model through questionnaires.ResultsThe results of the study highlight the different performance levels of different models in text embedding tasks. FastText has an F1-score of 46%, while the SBERT and OpenAI significantly outperform it, achieving F1-scores of 95 and 96%, respectively. The analysis highlights the effectiveness of LLMs, with GPT-4o achieving the highest F1-score of 95%, followed by Mistral and GPT-4o-mini with F1-scores of 94 and 92%, respectively. In addition, the performance ratings for the models are as follows: Mistral with 4.56, GPT-4o-mini with 4.45 and GPT-4o with 4.67. Among these, SBERT and Mistral are identified as the optimal choices due to their balanced performance, cost effectiveness, and ease of implementation.ConclusionThe RAGPR model can significantly improve the accuracy and personalization of web-based medical services, providing a scalable solution for improving patient-physician matching. |
| format | Article |
| id | doaj-art-9ec6c4b1a1c64555b4f4fe16ed481e4c |
| institution | Kabale University |
| issn | 2296-2565 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Public Health |
| spelling | doaj-art-9ec6c4b1a1c64555b4f4fe16ed481e4c2025-01-29T06:45:36ZengFrontiers Media S.A.Frontiers in Public Health2296-25652025-01-011310.3389/fpubh.2025.15014081501408Integrating retrieval-augmented generation for enhanced personalized physician recommendations in web-based medical services: model development studyYingbin Zheng0Yiwei Yan1Sai Chen2Yunping Cai3Kun Ren4Yishan Liu5Jiaying Zhuang6Min Zhao7Biomedical Big Data Center, The First Affiliated Hospital of Xiamen University, School of Medicine, Xiamen University, Xiamen, ChinaBiomedical Big Data Center, The First Affiliated Hospital of Xiamen University, School of Medicine, Xiamen University, Xiamen, ChinaMeteorological Disaster Prevention Technology Center, Xiamen Meteorological Bureau, Xiamen, ChinaMeteorological Disaster Prevention Technology Center, Xiamen Meteorological Bureau, Xiamen, ChinaMeteorological Disaster Prevention Technology Center, Xiamen Meteorological Bureau, Xiamen, ChinaSchool of Software Engineering, Taiyuan University of Technology, Taiyuan, ChinaBiomedical Big Data Center, The First Affiliated Hospital of Xiamen University, School of Medicine, Xiamen University, Xiamen, ChinaBiomedical Big Data Center, The First Affiliated Hospital of Xiamen University, School of Medicine, Xiamen University, Xiamen, ChinaBackgroundWeb-based medical services have significantly improved access to healthcare by enabling remote consultations, streamlining scheduling, and improving access to medical information. However, providing personalized physician recommendations remains a challenge, often relying on manual triage by schedulers, which can be limited by scalability and availability.ObjectiveThis study aimed to develop and validate a Retrieval-Augmented Generation-Based Physician Recommendation (RAGPR) model for better triage performance.MethodsThis study utilizes a comprehensive dataset consisting of 646,383 consultation records from the Internet Hospital of the First Affiliated Hospital of Xiamen University. The research primarily evaluates the performance of various embedding models, including FastText, SBERT, and OpenAI, for the purposes of clustering and classifying medical condition labels. Additionally, the study assesses the effectiveness of large language models (LLMs) by comparing Mistral, GPT-4o-mini, and GPT-4o. Furthermore, the study includes the participation of three triage staff members who contributed to the evaluation of the efficiency of the RAGPR model through questionnaires.ResultsThe results of the study highlight the different performance levels of different models in text embedding tasks. FastText has an F1-score of 46%, while the SBERT and OpenAI significantly outperform it, achieving F1-scores of 95 and 96%, respectively. The analysis highlights the effectiveness of LLMs, with GPT-4o achieving the highest F1-score of 95%, followed by Mistral and GPT-4o-mini with F1-scores of 94 and 92%, respectively. In addition, the performance ratings for the models are as follows: Mistral with 4.56, GPT-4o-mini with 4.45 and GPT-4o with 4.67. Among these, SBERT and Mistral are identified as the optimal choices due to their balanced performance, cost effectiveness, and ease of implementation.ConclusionThe RAGPR model can significantly improve the accuracy and personalization of web-based medical services, providing a scalable solution for improving patient-physician matching.https://www.frontiersin.org/articles/10.3389/fpubh.2025.1501408/fulllarge language modelsmistralSBERTtriage systemsretrieval-augmented generation-based physician recommendationRAGPR model |
| spellingShingle | Yingbin Zheng Yiwei Yan Sai Chen Yunping Cai Kun Ren Yishan Liu Jiaying Zhuang Min Zhao Integrating retrieval-augmented generation for enhanced personalized physician recommendations in web-based medical services: model development study Frontiers in Public Health large language models mistral SBERT triage systems retrieval-augmented generation-based physician recommendation RAGPR model |
| title | Integrating retrieval-augmented generation for enhanced personalized physician recommendations in web-based medical services: model development study |
| title_full | Integrating retrieval-augmented generation for enhanced personalized physician recommendations in web-based medical services: model development study |
| title_fullStr | Integrating retrieval-augmented generation for enhanced personalized physician recommendations in web-based medical services: model development study |
| title_full_unstemmed | Integrating retrieval-augmented generation for enhanced personalized physician recommendations in web-based medical services: model development study |
| title_short | Integrating retrieval-augmented generation for enhanced personalized physician recommendations in web-based medical services: model development study |
| title_sort | integrating retrieval augmented generation for enhanced personalized physician recommendations in web based medical services model development study |
| topic | large language models mistral SBERT triage systems retrieval-augmented generation-based physician recommendation RAGPR model |
| url | https://www.frontiersin.org/articles/10.3389/fpubh.2025.1501408/full |
| work_keys_str_mv | AT yingbinzheng integratingretrievalaugmentedgenerationforenhancedpersonalizedphysicianrecommendationsinwebbasedmedicalservicesmodeldevelopmentstudy AT yiweiyan integratingretrievalaugmentedgenerationforenhancedpersonalizedphysicianrecommendationsinwebbasedmedicalservicesmodeldevelopmentstudy AT saichen integratingretrievalaugmentedgenerationforenhancedpersonalizedphysicianrecommendationsinwebbasedmedicalservicesmodeldevelopmentstudy AT yunpingcai integratingretrievalaugmentedgenerationforenhancedpersonalizedphysicianrecommendationsinwebbasedmedicalservicesmodeldevelopmentstudy AT kunren integratingretrievalaugmentedgenerationforenhancedpersonalizedphysicianrecommendationsinwebbasedmedicalservicesmodeldevelopmentstudy AT yishanliu integratingretrievalaugmentedgenerationforenhancedpersonalizedphysicianrecommendationsinwebbasedmedicalservicesmodeldevelopmentstudy AT jiayingzhuang integratingretrievalaugmentedgenerationforenhancedpersonalizedphysicianrecommendationsinwebbasedmedicalservicesmodeldevelopmentstudy AT minzhao integratingretrievalaugmentedgenerationforenhancedpersonalizedphysicianrecommendationsinwebbasedmedicalservicesmodeldevelopmentstudy |