Development and Validation of a Machine Learning Model for Early Prediction of Delirium in Intensive Care Units Using Continuous Physiological Data: Retrospective Study
BackgroundDelirium in intensive care unit (ICU) patients poses a significant challenge, affecting patient outcomes and health care efficiency. Developing an accurate, real-time prediction model for delirium represents an advancement in critical care, addressing needs for time...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2025-04-01
|
| Series: | Journal of Medical Internet Research |
| Online Access: | https://www.jmir.org/2025/1/e59520 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850259546783612928 |
|---|---|
| author | Chanmin Park Changho Han Su Kyeong Jang Hyungjun Kim Sora Kim Byung Hee Kang Kyoungwon Jung Dukyong Yoon |
| author_facet | Chanmin Park Changho Han Su Kyeong Jang Hyungjun Kim Sora Kim Byung Hee Kang Kyoungwon Jung Dukyong Yoon |
| author_sort | Chanmin Park |
| collection | DOAJ |
| description |
BackgroundDelirium in intensive care unit (ICU) patients poses a significant challenge, affecting patient outcomes and health care efficiency. Developing an accurate, real-time prediction model for delirium represents an advancement in critical care, addressing needs for timely intervention and resource optimization in ICUs.
ObjectiveWe aimed to create a novel machine learning model for delirium prediction in ICU patients using only continuous physiological data.
MethodsWe developed models integrating routinely available clinical data, such as age, sex, and patient monitoring device outputs, to ensure practicality and adaptability in diverse clinical settings. To confirm the reliability of delirium determination records, we prospectively collected results of Confusion Assessment Method for the ICU (CAM-ICU) evaluations performed by qualified investigators from May 17, 2021, to December 23, 2022, determining Cohen κ coefficients. Participants were included in the study if they were aged ≥18 years at ICU admission, had delirium evaluations using the CAM-ICU, and had data collected for at least 4 hours before delirium diagnosis or nondiagnosis. The development cohort from Yongin Severance Hospital (March 1, 2020, to January 12, 2022) comprised 5478 records: 5129 (93.62%) records from 651 patients for training and 349 (6.37%) records from 163 patients for internal validation. For temporal validation, we used 4438 records from the same hospital (January 28, 2022, to December 31, 2022) to reflect potential seasonal variations. External validation was performed using data from 670 patients at Ajou University Hospital (March 2022 to September 2022). We evaluated machine learning algorithms (random forest [RF], extra-trees classifier, and light gradient boosting machine) and selected the RF model as the final model based on its performance. To confirm clinical utility, a decision curve analysis and temporal pattern for model prediction during the ICU stay were performed.
ResultsThe κ coefficient between labels generated by ICU nurses and prospectively verified by qualified researchers was 0.81, indicating reliable CAM-ICU results. Our final model showed robust performance in internal validation (area under the receiver operating characteristic curve [AUROC]: 0.82; area under the precision-recall curve [AUPRC]: 0.62) and maintained its accuracy in temporal validation (AUROC: 0.73; AUPRC: 0.85). External validation supported its effectiveness (AUROC: 0.84; AUPRC: 0.77). Decision curve analysis showed a positive net benefit at all thresholds, and the temporal pattern analysis showed a gradual increase in the model scores as the actual delirium diagnosis time approached.
ConclusionsWe developed a machine learning model for delirium prediction in ICU patients using routinely measured variables, including physiological waveforms. Our study demonstrates the potential of the RF model in predicting delirium, with consistent performance across various validation scenarios. The model uses noninvasive variables, making it applicable to a wide range of ICU patients, with minimal additional risk. |
| format | Article |
| id | doaj-art-88cb82cd2bfc4d5daacd2cbbf2dbc0cd |
| institution | OA Journals |
| issn | 1438-8871 |
| language | English |
| publishDate | 2025-04-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | Journal of Medical Internet Research |
| spelling | doaj-art-88cb82cd2bfc4d5daacd2cbbf2dbc0cd2025-08-20T01:55:50ZengJMIR PublicationsJournal of Medical Internet Research1438-88712025-04-0127e5952010.2196/59520Development and Validation of a Machine Learning Model for Early Prediction of Delirium in Intensive Care Units Using Continuous Physiological Data: Retrospective StudyChanmin Parkhttps://orcid.org/0000-0002-6088-6048Changho Hanhttps://orcid.org/0000-0003-4121-5465Su Kyeong Janghttps://orcid.org/0000-0002-0313-5491Hyungjun Kimhttps://orcid.org/0000-0002-4040-3715Sora Kimhttps://orcid.org/0000-0001-9787-2339Byung Hee Kanghttps://orcid.org/0000-0003-3204-3251Kyoungwon Junghttps://orcid.org/0000-0001-7895-0362Dukyong Yoonhttps://orcid.org/0000-0003-1635-8376 BackgroundDelirium in intensive care unit (ICU) patients poses a significant challenge, affecting patient outcomes and health care efficiency. Developing an accurate, real-time prediction model for delirium represents an advancement in critical care, addressing needs for timely intervention and resource optimization in ICUs. ObjectiveWe aimed to create a novel machine learning model for delirium prediction in ICU patients using only continuous physiological data. MethodsWe developed models integrating routinely available clinical data, such as age, sex, and patient monitoring device outputs, to ensure practicality and adaptability in diverse clinical settings. To confirm the reliability of delirium determination records, we prospectively collected results of Confusion Assessment Method for the ICU (CAM-ICU) evaluations performed by qualified investigators from May 17, 2021, to December 23, 2022, determining Cohen κ coefficients. Participants were included in the study if they were aged ≥18 years at ICU admission, had delirium evaluations using the CAM-ICU, and had data collected for at least 4 hours before delirium diagnosis or nondiagnosis. The development cohort from Yongin Severance Hospital (March 1, 2020, to January 12, 2022) comprised 5478 records: 5129 (93.62%) records from 651 patients for training and 349 (6.37%) records from 163 patients for internal validation. For temporal validation, we used 4438 records from the same hospital (January 28, 2022, to December 31, 2022) to reflect potential seasonal variations. External validation was performed using data from 670 patients at Ajou University Hospital (March 2022 to September 2022). We evaluated machine learning algorithms (random forest [RF], extra-trees classifier, and light gradient boosting machine) and selected the RF model as the final model based on its performance. To confirm clinical utility, a decision curve analysis and temporal pattern for model prediction during the ICU stay were performed. ResultsThe κ coefficient between labels generated by ICU nurses and prospectively verified by qualified researchers was 0.81, indicating reliable CAM-ICU results. Our final model showed robust performance in internal validation (area under the receiver operating characteristic curve [AUROC]: 0.82; area under the precision-recall curve [AUPRC]: 0.62) and maintained its accuracy in temporal validation (AUROC: 0.73; AUPRC: 0.85). External validation supported its effectiveness (AUROC: 0.84; AUPRC: 0.77). Decision curve analysis showed a positive net benefit at all thresholds, and the temporal pattern analysis showed a gradual increase in the model scores as the actual delirium diagnosis time approached. ConclusionsWe developed a machine learning model for delirium prediction in ICU patients using routinely measured variables, including physiological waveforms. Our study demonstrates the potential of the RF model in predicting delirium, with consistent performance across various validation scenarios. The model uses noninvasive variables, making it applicable to a wide range of ICU patients, with minimal additional risk.https://www.jmir.org/2025/1/e59520 |
| spellingShingle | Chanmin Park Changho Han Su Kyeong Jang Hyungjun Kim Sora Kim Byung Hee Kang Kyoungwon Jung Dukyong Yoon Development and Validation of a Machine Learning Model for Early Prediction of Delirium in Intensive Care Units Using Continuous Physiological Data: Retrospective Study Journal of Medical Internet Research |
| title | Development and Validation of a Machine Learning Model for Early Prediction of Delirium in Intensive Care Units Using Continuous Physiological Data: Retrospective Study |
| title_full | Development and Validation of a Machine Learning Model for Early Prediction of Delirium in Intensive Care Units Using Continuous Physiological Data: Retrospective Study |
| title_fullStr | Development and Validation of a Machine Learning Model for Early Prediction of Delirium in Intensive Care Units Using Continuous Physiological Data: Retrospective Study |
| title_full_unstemmed | Development and Validation of a Machine Learning Model for Early Prediction of Delirium in Intensive Care Units Using Continuous Physiological Data: Retrospective Study |
| title_short | Development and Validation of a Machine Learning Model for Early Prediction of Delirium in Intensive Care Units Using Continuous Physiological Data: Retrospective Study |
| title_sort | development and validation of a machine learning model for early prediction of delirium in intensive care units using continuous physiological data retrospective study |
| url | https://www.jmir.org/2025/1/e59520 |
| work_keys_str_mv | AT chanminpark developmentandvalidationofamachinelearningmodelforearlypredictionofdeliriuminintensivecareunitsusingcontinuousphysiologicaldataretrospectivestudy AT changhohan developmentandvalidationofamachinelearningmodelforearlypredictionofdeliriuminintensivecareunitsusingcontinuousphysiologicaldataretrospectivestudy AT sukyeongjang developmentandvalidationofamachinelearningmodelforearlypredictionofdeliriuminintensivecareunitsusingcontinuousphysiologicaldataretrospectivestudy AT hyungjunkim developmentandvalidationofamachinelearningmodelforearlypredictionofdeliriuminintensivecareunitsusingcontinuousphysiologicaldataretrospectivestudy AT sorakim developmentandvalidationofamachinelearningmodelforearlypredictionofdeliriuminintensivecareunitsusingcontinuousphysiologicaldataretrospectivestudy AT byungheekang developmentandvalidationofamachinelearningmodelforearlypredictionofdeliriuminintensivecareunitsusingcontinuousphysiologicaldataretrospectivestudy AT kyoungwonjung developmentandvalidationofamachinelearningmodelforearlypredictionofdeliriuminintensivecareunitsusingcontinuousphysiologicaldataretrospectivestudy AT dukyongyoon developmentandvalidationofamachinelearningmodelforearlypredictionofdeliriuminintensivecareunitsusingcontinuousphysiologicaldataretrospectivestudy |