Using Large Language Models to Retrieve Critical Data from Clinical Processes and Business Rules
Current clinical care relies heavily on complex, rule-based systems for tasks like diagnosis and treatment. However, these systems can be cumbersome and require constant updates. This study explores the potential of the large language model (LLM), LLaMA 2, to address these limitations. We tested LLa...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-12-01
|
| Series: | Bioengineering |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2306-5354/12/1/17 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832589090407579648 |
|---|---|
| author | Yunguo Yu Cesar A. Gomez-Cabello Svetlana Makarova Yogesh Parte Sahar Borna Syed Ali Haider Ariana Genovese Srinivasagam Prabha Antonio J. Forte |
| author_facet | Yunguo Yu Cesar A. Gomez-Cabello Svetlana Makarova Yogesh Parte Sahar Borna Syed Ali Haider Ariana Genovese Srinivasagam Prabha Antonio J. Forte |
| author_sort | Yunguo Yu |
| collection | DOAJ |
| description | Current clinical care relies heavily on complex, rule-based systems for tasks like diagnosis and treatment. However, these systems can be cumbersome and require constant updates. This study explores the potential of the large language model (LLM), LLaMA 2, to address these limitations. We tested LLaMA 2′s performance in interpreting complex clinical process models, such as Mayo Clinic Care Pathway Models (CPMs), and providing accurate clinical recommendations. LLM was trained on encoded pathways versions using DOT language, embedding them with SentenceTransformer, and then presented with hypothetical patient cases. We compared the token-level accuracy between LLM output and the ground truth by measuring both node and edge accuracy. LLaMA 2 accurately retrieved the diagnosis, suggested further evaluation, and delivered appropriate management steps, all based on the pathways. The average node accuracy across the different pathways was 0.91 (SD ± 0.045), while the average edge accuracy was 0.92 (SD ± 0.122). This study highlights the potential of LLMs for healthcare information retrieval, especially when relevant data are provided. Future research should focus on improving these models’ interpretability and their integration into existing clinical workflows. |
| format | Article |
| id | doaj-art-ffc6f38f95714e4b96ad41ecc5c6f17d |
| institution | Kabale University |
| issn | 2306-5354 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Bioengineering |
| spelling | doaj-art-ffc6f38f95714e4b96ad41ecc5c6f17d2025-01-24T13:22:58ZengMDPI AGBioengineering2306-53542024-12-011211710.3390/bioengineering12010017Using Large Language Models to Retrieve Critical Data from Clinical Processes and Business RulesYunguo Yu0Cesar A. Gomez-Cabello1Svetlana Makarova2Yogesh Parte3Sahar Borna4Syed Ali Haider5Ariana Genovese6Srinivasagam Prabha7Antonio J. Forte8Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USADivision of Plastic Surgery, Mayo Clinic, 4500 San Pablo Road, Jacksonville, FL 32224, USACenter for Digital Health, Mayo Clinic, Rochester, MN 55905, USACenter for Digital Health, Mayo Clinic, Rochester, MN 55905, USADivision of Plastic Surgery, Mayo Clinic, 4500 San Pablo Road, Jacksonville, FL 32224, USADivision of Plastic Surgery, Mayo Clinic, 4500 San Pablo Road, Jacksonville, FL 32224, USADivision of Plastic Surgery, Mayo Clinic, 4500 San Pablo Road, Jacksonville, FL 32224, USADivision of Plastic Surgery, Mayo Clinic, 4500 San Pablo Road, Jacksonville, FL 32224, USACenter for Digital Health, Mayo Clinic, Rochester, MN 55905, USACurrent clinical care relies heavily on complex, rule-based systems for tasks like diagnosis and treatment. However, these systems can be cumbersome and require constant updates. This study explores the potential of the large language model (LLM), LLaMA 2, to address these limitations. We tested LLaMA 2′s performance in interpreting complex clinical process models, such as Mayo Clinic Care Pathway Models (CPMs), and providing accurate clinical recommendations. LLM was trained on encoded pathways versions using DOT language, embedding them with SentenceTransformer, and then presented with hypothetical patient cases. We compared the token-level accuracy between LLM output and the ground truth by measuring both node and edge accuracy. LLaMA 2 accurately retrieved the diagnosis, suggested further evaluation, and delivered appropriate management steps, all based on the pathways. The average node accuracy across the different pathways was 0.91 (SD ± 0.045), while the average edge accuracy was 0.92 (SD ± 0.122). This study highlights the potential of LLMs for healthcare information retrieval, especially when relevant data are provided. Future research should focus on improving these models’ interpretability and their integration into existing clinical workflows.https://www.mdpi.com/2306-5354/12/1/17diagnosticsclinical decision supportArtificial Intelligencelarge language modelsdata retrieval |
| spellingShingle | Yunguo Yu Cesar A. Gomez-Cabello Svetlana Makarova Yogesh Parte Sahar Borna Syed Ali Haider Ariana Genovese Srinivasagam Prabha Antonio J. Forte Using Large Language Models to Retrieve Critical Data from Clinical Processes and Business Rules Bioengineering diagnostics clinical decision support Artificial Intelligence large language models data retrieval |
| title | Using Large Language Models to Retrieve Critical Data from Clinical Processes and Business Rules |
| title_full | Using Large Language Models to Retrieve Critical Data from Clinical Processes and Business Rules |
| title_fullStr | Using Large Language Models to Retrieve Critical Data from Clinical Processes and Business Rules |
| title_full_unstemmed | Using Large Language Models to Retrieve Critical Data from Clinical Processes and Business Rules |
| title_short | Using Large Language Models to Retrieve Critical Data from Clinical Processes and Business Rules |
| title_sort | using large language models to retrieve critical data from clinical processes and business rules |
| topic | diagnostics clinical decision support Artificial Intelligence large language models data retrieval |
| url | https://www.mdpi.com/2306-5354/12/1/17 |
| work_keys_str_mv | AT yunguoyu usinglargelanguagemodelstoretrievecriticaldatafromclinicalprocessesandbusinessrules AT cesaragomezcabello usinglargelanguagemodelstoretrievecriticaldatafromclinicalprocessesandbusinessrules AT svetlanamakarova usinglargelanguagemodelstoretrievecriticaldatafromclinicalprocessesandbusinessrules AT yogeshparte usinglargelanguagemodelstoretrievecriticaldatafromclinicalprocessesandbusinessrules AT saharborna usinglargelanguagemodelstoretrievecriticaldatafromclinicalprocessesandbusinessrules AT syedalihaider usinglargelanguagemodelstoretrievecriticaldatafromclinicalprocessesandbusinessrules AT arianagenovese usinglargelanguagemodelstoretrievecriticaldatafromclinicalprocessesandbusinessrules AT srinivasagamprabha usinglargelanguagemodelstoretrievecriticaldatafromclinicalprocessesandbusinessrules AT antoniojforte usinglargelanguagemodelstoretrievecriticaldatafromclinicalprocessesandbusinessrules |