Upskilling or deskilling? Measurable role of an AI-supported training for radiology residents: a lesson from the pandemic
Abstract Objectives This article aims to evaluate the use and effects of an artificial intelligence system supporting a critical diagnostic task during radiology resident training, addressing a research gap in this field. Materials and methods We involved eight residents evaluating 150 CXRs in three...
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2025-01-01
|
| Series: | Insights into Imaging |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s13244-024-01893-4 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832571596253954048 |
|---|---|
| author | Mattia Savardi Alberto Signoroni Sergio Benini Filippo Vaccher Matteo Alberti Pietro Ciolli Nunzia Di Meo Teresa Falcone Marco Ramanzin Barbara Romano Federica Sozzi Davide Farina |
| author_facet | Mattia Savardi Alberto Signoroni Sergio Benini Filippo Vaccher Matteo Alberti Pietro Ciolli Nunzia Di Meo Teresa Falcone Marco Ramanzin Barbara Romano Federica Sozzi Davide Farina |
| author_sort | Mattia Savardi |
| collection | DOAJ |
| description | Abstract Objectives This article aims to evaluate the use and effects of an artificial intelligence system supporting a critical diagnostic task during radiology resident training, addressing a research gap in this field. Materials and methods We involved eight residents evaluating 150 CXRs in three scenarios: no AI, on-demand AI, and integrated-AI. The considered task was the assessment of a multi-regional severity score of lung compromise in patients affected by COVID-19. The chosen artificial intelligence tool, fully integrated in the RIS/PACS, demonstrated superior performance in scoring compared to the average radiologist. Using quantitative metrics and questionnaires, we measured the ‘upskilling’ effects of using AI support and residents’ resilience to ‘deskilling,’ i.e., their ability to overcome AI errors. Results Residents required AI in 70% of cases when left free to choose. AI support significantly reduced severity score errors and increased inter-rater agreement by 22%. Residents were resilient to AI errors above an acceptability threshold. Questionnaires indicated high tool usefulness, reliability, and explainability, with a preference for collaborative AI scenarios. Conclusion With this work, we gathered quantitative and qualitative evidence of the beneficial use of a high-performance AI tool that is well integrated into the diagnostic workflow as a training aid for radiology residents. Critical relevance statement Balancing educational benefits and deskilling risks is essential to exploit AI systems as effective learning tools in radiology residency programs. Our work highlights metrics for evaluating these aspects. Key Points Insights into AI tools’ effects in radiology resident training are lacking. Metrics were defined to observe residents using an AI tool in different settings. This approach is advisable for evaluating AI tools in radiology training. Graphical Abstract |
| format | Article |
| id | doaj-art-8863c11ee1e64f23af9d09d4f7e309e8 |
| institution | Kabale University |
| issn | 1869-4101 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Insights into Imaging |
| spelling | doaj-art-8863c11ee1e64f23af9d09d4f7e309e82025-02-02T12:27:57ZengSpringerOpenInsights into Imaging1869-41012025-01-0116111110.1186/s13244-024-01893-4Upskilling or deskilling? Measurable role of an AI-supported training for radiology residents: a lesson from the pandemicMattia Savardi0Alberto Signoroni1Sergio Benini2Filippo Vaccher3Matteo Alberti4Pietro Ciolli5Nunzia Di Meo6Teresa Falcone7Marco Ramanzin8Barbara Romano9Federica Sozzi10Davide Farina11Department of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of BresciaDepartment of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of BresciaDepartment of Information Engineering, University of BresciaDepartment of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of BresciaDepartment of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of BresciaDepartment of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of BresciaDepartment of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of BresciaDepartment of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of BresciaDepartment of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of BresciaDepartment of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of BresciaDepartment of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of BresciaDepartment of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of BresciaAbstract Objectives This article aims to evaluate the use and effects of an artificial intelligence system supporting a critical diagnostic task during radiology resident training, addressing a research gap in this field. Materials and methods We involved eight residents evaluating 150 CXRs in three scenarios: no AI, on-demand AI, and integrated-AI. The considered task was the assessment of a multi-regional severity score of lung compromise in patients affected by COVID-19. The chosen artificial intelligence tool, fully integrated in the RIS/PACS, demonstrated superior performance in scoring compared to the average radiologist. Using quantitative metrics and questionnaires, we measured the ‘upskilling’ effects of using AI support and residents’ resilience to ‘deskilling,’ i.e., their ability to overcome AI errors. Results Residents required AI in 70% of cases when left free to choose. AI support significantly reduced severity score errors and increased inter-rater agreement by 22%. Residents were resilient to AI errors above an acceptability threshold. Questionnaires indicated high tool usefulness, reliability, and explainability, with a preference for collaborative AI scenarios. Conclusion With this work, we gathered quantitative and qualitative evidence of the beneficial use of a high-performance AI tool that is well integrated into the diagnostic workflow as a training aid for radiology residents. Critical relevance statement Balancing educational benefits and deskilling risks is essential to exploit AI systems as effective learning tools in radiology residency programs. Our work highlights metrics for evaluating these aspects. Key Points Insights into AI tools’ effects in radiology resident training are lacking. Metrics were defined to observe residents using an AI tool in different settings. This approach is advisable for evaluating AI tools in radiology training. Graphical Abstracthttps://doi.org/10.1186/s13244-024-01893-4Artificial intelligenceUpskilling-deskillingRadiology resident training |
| spellingShingle | Mattia Savardi Alberto Signoroni Sergio Benini Filippo Vaccher Matteo Alberti Pietro Ciolli Nunzia Di Meo Teresa Falcone Marco Ramanzin Barbara Romano Federica Sozzi Davide Farina Upskilling or deskilling? Measurable role of an AI-supported training for radiology residents: a lesson from the pandemic Insights into Imaging Artificial intelligence Upskilling-deskilling Radiology resident training |
| title | Upskilling or deskilling? Measurable role of an AI-supported training for radiology residents: a lesson from the pandemic |
| title_full | Upskilling or deskilling? Measurable role of an AI-supported training for radiology residents: a lesson from the pandemic |
| title_fullStr | Upskilling or deskilling? Measurable role of an AI-supported training for radiology residents: a lesson from the pandemic |
| title_full_unstemmed | Upskilling or deskilling? Measurable role of an AI-supported training for radiology residents: a lesson from the pandemic |
| title_short | Upskilling or deskilling? Measurable role of an AI-supported training for radiology residents: a lesson from the pandemic |
| title_sort | upskilling or deskilling measurable role of an ai supported training for radiology residents a lesson from the pandemic |
| topic | Artificial intelligence Upskilling-deskilling Radiology resident training |
| url | https://doi.org/10.1186/s13244-024-01893-4 |
| work_keys_str_mv | AT mattiasavardi upskillingordeskillingmeasurableroleofanaisupportedtrainingforradiologyresidentsalessonfromthepandemic AT albertosignoroni upskillingordeskillingmeasurableroleofanaisupportedtrainingforradiologyresidentsalessonfromthepandemic AT sergiobenini upskillingordeskillingmeasurableroleofanaisupportedtrainingforradiologyresidentsalessonfromthepandemic AT filippovaccher upskillingordeskillingmeasurableroleofanaisupportedtrainingforradiologyresidentsalessonfromthepandemic AT matteoalberti upskillingordeskillingmeasurableroleofanaisupportedtrainingforradiologyresidentsalessonfromthepandemic AT pietrociolli upskillingordeskillingmeasurableroleofanaisupportedtrainingforradiologyresidentsalessonfromthepandemic AT nunziadimeo upskillingordeskillingmeasurableroleofanaisupportedtrainingforradiologyresidentsalessonfromthepandemic AT teresafalcone upskillingordeskillingmeasurableroleofanaisupportedtrainingforradiologyresidentsalessonfromthepandemic AT marcoramanzin upskillingordeskillingmeasurableroleofanaisupportedtrainingforradiologyresidentsalessonfromthepandemic AT barbararomano upskillingordeskillingmeasurableroleofanaisupportedtrainingforradiologyresidentsalessonfromthepandemic AT federicasozzi upskillingordeskillingmeasurableroleofanaisupportedtrainingforradiologyresidentsalessonfromthepandemic AT davidefarina upskillingordeskillingmeasurableroleofanaisupportedtrainingforradiologyresidentsalessonfromthepandemic |