Discrimination of Radiologists' Experience Level Using Eye-Tracking Technology and Machine Learning: Case Study
BackgroundPerception-related errors comprise most diagnostic mistakes in radiology. To mitigate this problem, radiologists use personalized and high-dimensional visual search strategies, otherwise known as search patterns. Qualitative descriptions of these search patterns, wh...
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2025-01-01
|
| Series: | JMIR Formative Research |
| Online Access: | https://formative.jmir.org/2025/1/e53928 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832591168774340608 |
|---|---|
| author | Stanford Martinez Carolina Ramirez-Tamayo Syed Hasib Akhter Faruqui Kal Clark Adel Alaeddini Nicholas Czarnek Aarushi Aggarwal Sahra Emamzadeh Jeffrey R Mock Edward J Golob |
| author_facet | Stanford Martinez Carolina Ramirez-Tamayo Syed Hasib Akhter Faruqui Kal Clark Adel Alaeddini Nicholas Czarnek Aarushi Aggarwal Sahra Emamzadeh Jeffrey R Mock Edward J Golob |
| author_sort | Stanford Martinez |
| collection | DOAJ |
| description |
BackgroundPerception-related errors comprise most diagnostic mistakes in radiology. To mitigate this problem, radiologists use personalized and high-dimensional visual search strategies, otherwise known as search patterns. Qualitative descriptions of these search patterns, which involve the physician verbalizing or annotating the order he or she analyzes the image, can be unreliable due to discrepancies in what is reported versus the actual visual patterns. This discrepancy can interfere with quality improvement interventions and negatively impact patient care.
ObjectiveThe objective of this study is to provide an alternative method for distinguishing between radiologists by means of captured eye-tracking data such that the raw gaze (or processed fixation data) can be used to discriminate users based on subconscious behavior in visual inspection.
MethodsWe present a novel discretized feature encoding based on spatiotemporal binning of fixation data for efficient geometric alignment and temporal ordering of eye movement when reading chest x-rays. The encoded features of the eye-fixation data are used by machine learning classifiers to discriminate between faculty and trainee radiologists. A clinical trial case study was conducted using metrics such as the area under the curve, accuracy, F1-score, sensitivity, and specificity to evaluate the discriminability between the 2 groups regarding their level of experience. The classification performance was then compared with state-of-the-art methodologies. In addition, a repeatability experiment using a separate dataset, experimental protocol, and eye tracker was performed with 8 participants to evaluate the robustness of the proposed approach.
ResultsThe numerical results from both experiments demonstrate that classifiers using the proposed feature encoding methods outperform the current state-of-the-art in differentiating between radiologists in terms of experience level. An average performance gain of 6.9% is observed compared with traditional features while classifying experience levels of radiologists. This gain in accuracy is also substantial across different eye tracker–collected datasets, with improvements of 6.41% using the Tobii eye tracker and 7.29% using the EyeLink eye tracker. These results signify the potential impact of the proposed method for identifying radiologists’ level of expertise and those who would benefit from additional training.
ConclusionsThe effectiveness of the proposed spatiotemporal discretization approach, validated across diverse datasets and various classification metrics, underscores its potential for objective evaluation, informing targeted interventions and training strategies in radiology. This research advances reliable assessment tools, addressing challenges in perception-related errors to enhance patient care outcomes. |
| format | Article |
| id | doaj-art-6fb9275ee1da46d1820d77805060f965 |
| institution | Kabale University |
| issn | 2561-326X |
| language | English |
| publishDate | 2025-01-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Formative Research |
| spelling | doaj-art-6fb9275ee1da46d1820d77805060f9652025-01-22T18:15:26ZengJMIR PublicationsJMIR Formative Research2561-326X2025-01-019e5392810.2196/53928Discrimination of Radiologists' Experience Level Using Eye-Tracking Technology and Machine Learning: Case StudyStanford Martinezhttps://orcid.org/0000-0001-7735-6572Carolina Ramirez-Tamayohttps://orcid.org/0000-0002-4413-1366Syed Hasib Akhter Faruquihttps://orcid.org/0000-0002-5073-8690Kal Clarkhttps://orcid.org/0000-0002-2511-7577Adel Alaeddinihttps://orcid.org/0000-0003-4451-3150Nicholas Czarnekhttps://orcid.org/0000-0002-7889-0184Aarushi Aggarwalhttps://orcid.org/0000-0001-6557-680XSahra Emamzadehhttps://orcid.org/0000-0002-1053-1441Jeffrey R Mockhttps://orcid.org/0000-0003-0446-6687Edward J Golobhttps://orcid.org/0000-0002-1560-9076 BackgroundPerception-related errors comprise most diagnostic mistakes in radiology. To mitigate this problem, radiologists use personalized and high-dimensional visual search strategies, otherwise known as search patterns. Qualitative descriptions of these search patterns, which involve the physician verbalizing or annotating the order he or she analyzes the image, can be unreliable due to discrepancies in what is reported versus the actual visual patterns. This discrepancy can interfere with quality improvement interventions and negatively impact patient care. ObjectiveThe objective of this study is to provide an alternative method for distinguishing between radiologists by means of captured eye-tracking data such that the raw gaze (or processed fixation data) can be used to discriminate users based on subconscious behavior in visual inspection. MethodsWe present a novel discretized feature encoding based on spatiotemporal binning of fixation data for efficient geometric alignment and temporal ordering of eye movement when reading chest x-rays. The encoded features of the eye-fixation data are used by machine learning classifiers to discriminate between faculty and trainee radiologists. A clinical trial case study was conducted using metrics such as the area under the curve, accuracy, F1-score, sensitivity, and specificity to evaluate the discriminability between the 2 groups regarding their level of experience. The classification performance was then compared with state-of-the-art methodologies. In addition, a repeatability experiment using a separate dataset, experimental protocol, and eye tracker was performed with 8 participants to evaluate the robustness of the proposed approach. ResultsThe numerical results from both experiments demonstrate that classifiers using the proposed feature encoding methods outperform the current state-of-the-art in differentiating between radiologists in terms of experience level. An average performance gain of 6.9% is observed compared with traditional features while classifying experience levels of radiologists. This gain in accuracy is also substantial across different eye tracker–collected datasets, with improvements of 6.41% using the Tobii eye tracker and 7.29% using the EyeLink eye tracker. These results signify the potential impact of the proposed method for identifying radiologists’ level of expertise and those who would benefit from additional training. ConclusionsThe effectiveness of the proposed spatiotemporal discretization approach, validated across diverse datasets and various classification metrics, underscores its potential for objective evaluation, informing targeted interventions and training strategies in radiology. This research advances reliable assessment tools, addressing challenges in perception-related errors to enhance patient care outcomes.https://formative.jmir.org/2025/1/e53928 |
| spellingShingle | Stanford Martinez Carolina Ramirez-Tamayo Syed Hasib Akhter Faruqui Kal Clark Adel Alaeddini Nicholas Czarnek Aarushi Aggarwal Sahra Emamzadeh Jeffrey R Mock Edward J Golob Discrimination of Radiologists' Experience Level Using Eye-Tracking Technology and Machine Learning: Case Study JMIR Formative Research |
| title | Discrimination of Radiologists' Experience Level Using Eye-Tracking Technology and Machine Learning: Case Study |
| title_full | Discrimination of Radiologists' Experience Level Using Eye-Tracking Technology and Machine Learning: Case Study |
| title_fullStr | Discrimination of Radiologists' Experience Level Using Eye-Tracking Technology and Machine Learning: Case Study |
| title_full_unstemmed | Discrimination of Radiologists' Experience Level Using Eye-Tracking Technology and Machine Learning: Case Study |
| title_short | Discrimination of Radiologists' Experience Level Using Eye-Tracking Technology and Machine Learning: Case Study |
| title_sort | discrimination of radiologists experience level using eye tracking technology and machine learning case study |
| url | https://formative.jmir.org/2025/1/e53928 |
| work_keys_str_mv | AT stanfordmartinez discriminationofradiologistsexperiencelevelusingeyetrackingtechnologyandmachinelearningcasestudy AT carolinaramireztamayo discriminationofradiologistsexperiencelevelusingeyetrackingtechnologyandmachinelearningcasestudy AT syedhasibakhterfaruqui discriminationofradiologistsexperiencelevelusingeyetrackingtechnologyandmachinelearningcasestudy AT kalclark discriminationofradiologistsexperiencelevelusingeyetrackingtechnologyandmachinelearningcasestudy AT adelalaeddini discriminationofradiologistsexperiencelevelusingeyetrackingtechnologyandmachinelearningcasestudy AT nicholasczarnek discriminationofradiologistsexperiencelevelusingeyetrackingtechnologyandmachinelearningcasestudy AT aarushiaggarwal discriminationofradiologistsexperiencelevelusingeyetrackingtechnologyandmachinelearningcasestudy AT sahraemamzadeh discriminationofradiologistsexperiencelevelusingeyetrackingtechnologyandmachinelearningcasestudy AT jeffreyrmock discriminationofradiologistsexperiencelevelusingeyetrackingtechnologyandmachinelearningcasestudy AT edwardjgolob discriminationofradiologistsexperiencelevelusingeyetrackingtechnologyandmachinelearningcasestudy |