Use of computer vision analysis for labeling inattention periods in EEG recordings with visual stimuli
Abstract Electroencephalography (EEG) recordings with visual stimuli require detailed coding to determine the periods of participant’s attention. Here we propose to use a supervised machine learning model and off-the-shelf video cameras only. We extract computer vision-based features such as head po...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-08-01
|
| Series: | Scientific Reports |
| Subjects: | |
| Online Access: | https://doi.org/10.1038/s41598-025-10511-2 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849226192180215808 |
|---|---|
| author | Dmitry Yu Isaev Samantha Major Kimberly L. H. Carpenter Jordan Grapel Zhuoqing Chang Matias Di Martino David Carlson Geraldine Dawson Guillermo Sapiro |
| author_facet | Dmitry Yu Isaev Samantha Major Kimberly L. H. Carpenter Jordan Grapel Zhuoqing Chang Matias Di Martino David Carlson Geraldine Dawson Guillermo Sapiro |
| author_sort | Dmitry Yu Isaev |
| collection | DOAJ |
| description | Abstract Electroencephalography (EEG) recordings with visual stimuli require detailed coding to determine the periods of participant’s attention. Here we propose to use a supervised machine learning model and off-the-shelf video cameras only. We extract computer vision-based features such as head pose, gaze, and face landmarks from the video of the participant, and train the machine learning model (multi-layer perceptron) on an initial dataset, then adapt it with a small subset of data from a new participant. Using a sample size of 23 autistic children with and without co-occurring ADHD (attention-deficit/hyperactivity disorder) aged 49–95 months, and training on additional 2560 labeled frames (equivalent to 85.3 s of the video) of a new participant, the median area under the receiver operating characteristic curve for inattention detection was 0.989 (IQR 0.984–0.993) and the median inter-rater reliability (Cohen’s kappa) with a trained human annotator was 0.888. Agreement with human annotations for nine participants was in the 0.616–0.944 range. Our results demonstrate the feasibility of automatic tools to detect inattention during EEG recordings, and its potential to reduce the subjectivity and time burden of human attention coding. The tool for model adaptation and visualization of the computer vision features is made publicly available to the research community. |
| format | Article |
| id | doaj-art-2b03593f7fc94de4bce73a6f82eb3d00 |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-08-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-2b03593f7fc94de4bce73a6f82eb3d002025-08-24T11:30:57ZengNature PortfolioScientific Reports2045-23222025-08-0115111110.1038/s41598-025-10511-2Use of computer vision analysis for labeling inattention periods in EEG recordings with visual stimuliDmitry Yu Isaev0Samantha Major1Kimberly L. H. Carpenter2Jordan Grapel3Zhuoqing Chang4Matias Di Martino5David Carlson6Geraldine Dawson7Guillermo Sapiro8Department of Electrical and Computer Engineering, Duke UniversityDuke Center for Autism and Brain Development, Department of Psychiatry and Behavioral Sciences, Duke University School of MedicineDuke Center for Autism and Brain Development, Department of Psychiatry and Behavioral Sciences, Duke University School of MedicineDuke Center for Autism and Brain Development, Department of Psychiatry and Behavioral Sciences, Duke University School of MedicineDepartment of Electrical and Computer Engineering, Duke UniversityUniversidad Católica del UruguayDepartments of Civil and Environmental Engineering, Biostatistics and Bioinformatics, Duke UniversityDuke Center for Autism and Brain Development, Department of Psychiatry and Behavioral Sciences, Duke University School of MedicineDepartment of Electrical and Computer Engineering, Duke UniversityAbstract Electroencephalography (EEG) recordings with visual stimuli require detailed coding to determine the periods of participant’s attention. Here we propose to use a supervised machine learning model and off-the-shelf video cameras only. We extract computer vision-based features such as head pose, gaze, and face landmarks from the video of the participant, and train the machine learning model (multi-layer perceptron) on an initial dataset, then adapt it with a small subset of data from a new participant. Using a sample size of 23 autistic children with and without co-occurring ADHD (attention-deficit/hyperactivity disorder) aged 49–95 months, and training on additional 2560 labeled frames (equivalent to 85.3 s of the video) of a new participant, the median area under the receiver operating characteristic curve for inattention detection was 0.989 (IQR 0.984–0.993) and the median inter-rater reliability (Cohen’s kappa) with a trained human annotator was 0.888. Agreement with human annotations for nine participants was in the 0.616–0.944 range. Our results demonstrate the feasibility of automatic tools to detect inattention during EEG recordings, and its potential to reduce the subjectivity and time burden of human attention coding. The tool for model adaptation and visualization of the computer vision features is made publicly available to the research community.https://doi.org/10.1038/s41598-025-10511-2EEGVisual attentionComputer visionMachine learningData processing automation |
| spellingShingle | Dmitry Yu Isaev Samantha Major Kimberly L. H. Carpenter Jordan Grapel Zhuoqing Chang Matias Di Martino David Carlson Geraldine Dawson Guillermo Sapiro Use of computer vision analysis for labeling inattention periods in EEG recordings with visual stimuli Scientific Reports EEG Visual attention Computer vision Machine learning Data processing automation |
| title | Use of computer vision analysis for labeling inattention periods in EEG recordings with visual stimuli |
| title_full | Use of computer vision analysis for labeling inattention periods in EEG recordings with visual stimuli |
| title_fullStr | Use of computer vision analysis for labeling inattention periods in EEG recordings with visual stimuli |
| title_full_unstemmed | Use of computer vision analysis for labeling inattention periods in EEG recordings with visual stimuli |
| title_short | Use of computer vision analysis for labeling inattention periods in EEG recordings with visual stimuli |
| title_sort | use of computer vision analysis for labeling inattention periods in eeg recordings with visual stimuli |
| topic | EEG Visual attention Computer vision Machine learning Data processing automation |
| url | https://doi.org/10.1038/s41598-025-10511-2 |
| work_keys_str_mv | AT dmitryyuisaev useofcomputervisionanalysisforlabelinginattentionperiodsineegrecordingswithvisualstimuli AT samanthamajor useofcomputervisionanalysisforlabelinginattentionperiodsineegrecordingswithvisualstimuli AT kimberlylhcarpenter useofcomputervisionanalysisforlabelinginattentionperiodsineegrecordingswithvisualstimuli AT jordangrapel useofcomputervisionanalysisforlabelinginattentionperiodsineegrecordingswithvisualstimuli AT zhuoqingchang useofcomputervisionanalysisforlabelinginattentionperiodsineegrecordingswithvisualstimuli AT matiasdimartino useofcomputervisionanalysisforlabelinginattentionperiodsineegrecordingswithvisualstimuli AT davidcarlson useofcomputervisionanalysisforlabelinginattentionperiodsineegrecordingswithvisualstimuli AT geraldinedawson useofcomputervisionanalysisforlabelinginattentionperiodsineegrecordingswithvisualstimuli AT guillermosapiro useofcomputervisionanalysisforlabelinginattentionperiodsineegrecordingswithvisualstimuli |