AI for IMPACTS Framework for Evaluating the Long-Term Real-World Impacts of AI-Powered Clinician Tools: Systematic Review and Narrative Synthesis
BackgroundArtificial intelligence (AI) has the potential to revolutionize health care by enhancing both clinical outcomes and operational efficiency. However, its clinical adoption has been slower than anticipated, largely due to the absence of comprehensive evaluation framew...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2025-02-01
|
| Series: | Journal of Medical Internet Research |
| Online Access: | https://www.jmir.org/2025/1/e67485 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832095327075696640 |
|---|---|
| author | Christine Jacob Noé Brasier Emanuele Laurenzi Sabina Heuss Stavroula-Georgia Mougiakakou Arzu Cöltekin Marc K Peter |
| author_facet | Christine Jacob Noé Brasier Emanuele Laurenzi Sabina Heuss Stavroula-Georgia Mougiakakou Arzu Cöltekin Marc K Peter |
| author_sort | Christine Jacob |
| collection | DOAJ |
| description |
BackgroundArtificial intelligence (AI) has the potential to revolutionize health care by enhancing both clinical outcomes and operational efficiency. However, its clinical adoption has been slower than anticipated, largely due to the absence of comprehensive evaluation frameworks. Existing frameworks remain insufficient and tend to emphasize technical metrics such as accuracy and validation, while overlooking critical real-world factors such as clinical impact, integration, and economic sustainability. This narrow focus prevents AI tools from being effectively implemented, limiting their broader impact and long-term viability in clinical practice.
ObjectiveThis study aimed to create a framework for assessing AI in health care, extending beyond technical metrics to incorporate social and organizational dimensions. The framework was developed by systematically reviewing, analyzing, and synthesizing the evaluation criteria necessary for successful implementation, focusing on the long-term real-world impact of AI in clinical practice.
MethodsA search was performed in July 2024 across the PubMed, Cochrane, Scopus, and IEEE Xplore databases to identify relevant studies published in English between January 2019 and mid-July 2024, yielding 3528 results, among which 44 studies met the inclusion criteria. The systematic review followed PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines and the Cochrane Handbook for Systematic Reviews. Data were analyzed using NVivo through thematic analysis and narrative synthesis to identify key emergent themes in the studies.
ResultsBy synthesizing the included studies, we developed a framework that goes beyond the traditional focus on technical metrics or study-level methodologies. It integrates clinical context and real-world implementation factors, offering a more comprehensive approach to evaluating AI tools. With our focus on assessing the long-term real-world impact of AI technologies in health care, we named the framework AI for IMPACTS. The criteria are organized into seven key clusters, each corresponding to a letter in the acronym: (1) I—integration, interoperability, and workflow; (2) M—monitoring, governance, and accountability; (3) P—performance and quality metrics; (4) A—acceptability, trust, and training; (5) C—cost and economic evaluation; (6) T—technological safety and transparency; and (7) S—scalability and impact. These are further broken down into 28 specific subcriteria.
ConclusionsThe AI for IMPACTS framework offers a holistic approach to evaluate the long-term real-world impact of AI tools in the heterogeneous and challenging health care context and lays the groundwork for further validation through expert consensus and testing of the framework in real-world health care settings. It is important to emphasize that multidisciplinary expertise is essential for assessment, yet many assessors lack the necessary training. In addition, traditional evaluation methods struggle to keep pace with AI’s rapid development. To ensure successful AI integration, flexible, fast-tracked assessment processes and proper assessor training are needed to maintain rigorous standards while adapting to AI’s dynamic evolution.
Trial Registrationreviewregistry1859; https://tinyurl.com/ysn2d7sh |
| format | Article |
| id | doaj-art-604b870221ec408ab4d1f95f4e3eb657 |
| institution | Kabale University |
| issn | 1438-8871 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | Journal of Medical Internet Research |
| spelling | doaj-art-604b870221ec408ab4d1f95f4e3eb6572025-02-05T22:01:40ZengJMIR PublicationsJournal of Medical Internet Research1438-88712025-02-0127e6748510.2196/67485AI for IMPACTS Framework for Evaluating the Long-Term Real-World Impacts of AI-Powered Clinician Tools: Systematic Review and Narrative SynthesisChristine Jacobhttps://orcid.org/0000-0002-8817-0148Noé Brasierhttps://orcid.org/0000-0003-0186-0865Emanuele Laurenzihttps://orcid.org/0000-0001-9142-7488Sabina Heusshttps://orcid.org/0000-0002-1171-918XStavroula-Georgia Mougiakakouhttps://orcid.org/0000-0002-6355-9982Arzu Cöltekinhttps://orcid.org/0000-0002-3178-3509Marc K Peterhttps://orcid.org/0000-0002-2897-0389 BackgroundArtificial intelligence (AI) has the potential to revolutionize health care by enhancing both clinical outcomes and operational efficiency. However, its clinical adoption has been slower than anticipated, largely due to the absence of comprehensive evaluation frameworks. Existing frameworks remain insufficient and tend to emphasize technical metrics such as accuracy and validation, while overlooking critical real-world factors such as clinical impact, integration, and economic sustainability. This narrow focus prevents AI tools from being effectively implemented, limiting their broader impact and long-term viability in clinical practice. ObjectiveThis study aimed to create a framework for assessing AI in health care, extending beyond technical metrics to incorporate social and organizational dimensions. The framework was developed by systematically reviewing, analyzing, and synthesizing the evaluation criteria necessary for successful implementation, focusing on the long-term real-world impact of AI in clinical practice. MethodsA search was performed in July 2024 across the PubMed, Cochrane, Scopus, and IEEE Xplore databases to identify relevant studies published in English between January 2019 and mid-July 2024, yielding 3528 results, among which 44 studies met the inclusion criteria. The systematic review followed PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines and the Cochrane Handbook for Systematic Reviews. Data were analyzed using NVivo through thematic analysis and narrative synthesis to identify key emergent themes in the studies. ResultsBy synthesizing the included studies, we developed a framework that goes beyond the traditional focus on technical metrics or study-level methodologies. It integrates clinical context and real-world implementation factors, offering a more comprehensive approach to evaluating AI tools. With our focus on assessing the long-term real-world impact of AI technologies in health care, we named the framework AI for IMPACTS. The criteria are organized into seven key clusters, each corresponding to a letter in the acronym: (1) I—integration, interoperability, and workflow; (2) M—monitoring, governance, and accountability; (3) P—performance and quality metrics; (4) A—acceptability, trust, and training; (5) C—cost and economic evaluation; (6) T—technological safety and transparency; and (7) S—scalability and impact. These are further broken down into 28 specific subcriteria. ConclusionsThe AI for IMPACTS framework offers a holistic approach to evaluate the long-term real-world impact of AI tools in the heterogeneous and challenging health care context and lays the groundwork for further validation through expert consensus and testing of the framework in real-world health care settings. It is important to emphasize that multidisciplinary expertise is essential for assessment, yet many assessors lack the necessary training. In addition, traditional evaluation methods struggle to keep pace with AI’s rapid development. To ensure successful AI integration, flexible, fast-tracked assessment processes and proper assessor training are needed to maintain rigorous standards while adapting to AI’s dynamic evolution. Trial Registrationreviewregistry1859; https://tinyurl.com/ysn2d7shhttps://www.jmir.org/2025/1/e67485 |
| spellingShingle | Christine Jacob Noé Brasier Emanuele Laurenzi Sabina Heuss Stavroula-Georgia Mougiakakou Arzu Cöltekin Marc K Peter AI for IMPACTS Framework for Evaluating the Long-Term Real-World Impacts of AI-Powered Clinician Tools: Systematic Review and Narrative Synthesis Journal of Medical Internet Research |
| title | AI for IMPACTS Framework for Evaluating the Long-Term Real-World Impacts of AI-Powered Clinician Tools: Systematic Review and Narrative Synthesis |
| title_full | AI for IMPACTS Framework for Evaluating the Long-Term Real-World Impacts of AI-Powered Clinician Tools: Systematic Review and Narrative Synthesis |
| title_fullStr | AI for IMPACTS Framework for Evaluating the Long-Term Real-World Impacts of AI-Powered Clinician Tools: Systematic Review and Narrative Synthesis |
| title_full_unstemmed | AI for IMPACTS Framework for Evaluating the Long-Term Real-World Impacts of AI-Powered Clinician Tools: Systematic Review and Narrative Synthesis |
| title_short | AI for IMPACTS Framework for Evaluating the Long-Term Real-World Impacts of AI-Powered Clinician Tools: Systematic Review and Narrative Synthesis |
| title_sort | ai for impacts framework for evaluating the long term real world impacts of ai powered clinician tools systematic review and narrative synthesis |
| url | https://www.jmir.org/2025/1/e67485 |
| work_keys_str_mv | AT christinejacob aiforimpactsframeworkforevaluatingthelongtermrealworldimpactsofaipoweredcliniciantoolssystematicreviewandnarrativesynthesis AT noebrasier aiforimpactsframeworkforevaluatingthelongtermrealworldimpactsofaipoweredcliniciantoolssystematicreviewandnarrativesynthesis AT emanuelelaurenzi aiforimpactsframeworkforevaluatingthelongtermrealworldimpactsofaipoweredcliniciantoolssystematicreviewandnarrativesynthesis AT sabinaheuss aiforimpactsframeworkforevaluatingthelongtermrealworldimpactsofaipoweredcliniciantoolssystematicreviewandnarrativesynthesis AT stavroulageorgiamougiakakou aiforimpactsframeworkforevaluatingthelongtermrealworldimpactsofaipoweredcliniciantoolssystematicreviewandnarrativesynthesis AT arzucoltekin aiforimpactsframeworkforevaluatingthelongtermrealworldimpactsofaipoweredcliniciantoolssystematicreviewandnarrativesynthesis AT marckpeter aiforimpactsframeworkforevaluatingthelongtermrealworldimpactsofaipoweredcliniciantoolssystematicreviewandnarrativesynthesis |