Extensive benchmarking of a method that estimates external model performance from limited statistical characteristics
Abstract Predictive model performance may deteriorate when applied to data sources that were not used for training, thus, external validation is a key step in successful model deployment. As access to patient-level external data sources is typically limited, we recently proposed a method that estima...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-01-01
|
| Series: | npj Digital Medicine |
| Online Access: | https://doi.org/10.1038/s41746-024-01414-z |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832571303306985472 |
|---|---|
| author | Tal El-Hay Jenna M. Reps Chen Yanover |
| author_facet | Tal El-Hay Jenna M. Reps Chen Yanover |
| author_sort | Tal El-Hay |
| collection | DOAJ |
| description | Abstract Predictive model performance may deteriorate when applied to data sources that were not used for training, thus, external validation is a key step in successful model deployment. As access to patient-level external data sources is typically limited, we recently proposed a method that estimates external model performance using only external summary statistics. Here, we benchmark the proposed method on multiple tasks using five large heterogeneous US data sources, where each, in turn, plays the role of an internal source and the remaining—external. Results showed accurate estimations for all metrics: 95th error percentiles for the area under the receiver operating characteristics (discrimination), calibration-in-the-large (calibration), Brier and scaled Brier scores (overall accuracy) of 0.03, 0.08, 0.0002, and 0.07, respectively. These results demonstrate the feasibility of estimating the transportability of prediction models using an internal cohort and external statistics. It may become an important accelerator of model deployment. |
| format | Article |
| id | doaj-art-5b356e3c50d543f3bca891aa43557fb0 |
| institution | Kabale University |
| issn | 2398-6352 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | npj Digital Medicine |
| spelling | doaj-art-5b356e3c50d543f3bca891aa43557fb02025-02-02T12:43:40ZengNature Portfolionpj Digital Medicine2398-63522025-01-018111010.1038/s41746-024-01414-zExtensive benchmarking of a method that estimates external model performance from limited statistical characteristicsTal El-Hay0Jenna M. Reps1Chen Yanover2KI Research InstituteJanssen Research and DevelopmentKI Research InstituteAbstract Predictive model performance may deteriorate when applied to data sources that were not used for training, thus, external validation is a key step in successful model deployment. As access to patient-level external data sources is typically limited, we recently proposed a method that estimates external model performance using only external summary statistics. Here, we benchmark the proposed method on multiple tasks using five large heterogeneous US data sources, where each, in turn, plays the role of an internal source and the remaining—external. Results showed accurate estimations for all metrics: 95th error percentiles for the area under the receiver operating characteristics (discrimination), calibration-in-the-large (calibration), Brier and scaled Brier scores (overall accuracy) of 0.03, 0.08, 0.0002, and 0.07, respectively. These results demonstrate the feasibility of estimating the transportability of prediction models using an internal cohort and external statistics. It may become an important accelerator of model deployment.https://doi.org/10.1038/s41746-024-01414-z |
| spellingShingle | Tal El-Hay Jenna M. Reps Chen Yanover Extensive benchmarking of a method that estimates external model performance from limited statistical characteristics npj Digital Medicine |
| title | Extensive benchmarking of a method that estimates external model performance from limited statistical characteristics |
| title_full | Extensive benchmarking of a method that estimates external model performance from limited statistical characteristics |
| title_fullStr | Extensive benchmarking of a method that estimates external model performance from limited statistical characteristics |
| title_full_unstemmed | Extensive benchmarking of a method that estimates external model performance from limited statistical characteristics |
| title_short | Extensive benchmarking of a method that estimates external model performance from limited statistical characteristics |
| title_sort | extensive benchmarking of a method that estimates external model performance from limited statistical characteristics |
| url | https://doi.org/10.1038/s41746-024-01414-z |
| work_keys_str_mv | AT talelhay extensivebenchmarkingofamethodthatestimatesexternalmodelperformancefromlimitedstatisticalcharacteristics AT jennamreps extensivebenchmarkingofamethodthatestimatesexternalmodelperformancefromlimitedstatisticalcharacteristics AT chenyanover extensivebenchmarkingofamethodthatestimatesexternalmodelperformancefromlimitedstatisticalcharacteristics |