Random Cross-Validation Produces Biased Assessment of Machine Learning Performance in Regional Landslide Susceptibility Prediction
Machine learning (ML) models are extensively used in spatial predictive modeling, including landslide susceptibility prediction. The performance statistics of these models are vital for assessing their reliability, which is typically obtained using the random cross-validation (R-CV) method. However,...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-01-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/17/2/213 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832587594753376256 |
|---|---|
| author | Chandan Kumar Gabriel Walton Paul Santi Carlos Luza |
| author_facet | Chandan Kumar Gabriel Walton Paul Santi Carlos Luza |
| author_sort | Chandan Kumar |
| collection | DOAJ |
| description | Machine learning (ML) models are extensively used in spatial predictive modeling, including landslide susceptibility prediction. The performance statistics of these models are vital for assessing their reliability, which is typically obtained using the random cross-validation (R-CV) method. However, R-CV has a major drawback, i.e., it ignores the spatial autocorrelation (SAC) inherent in spatial datasets when partitioning the training and testing sets. We assessed the impact of SAC at three crucial phases of ML modeling: hyperparameter tuning, performance evaluation, and learning curve analysis. As an alternative to R-CV, we used spatial cross-validation (S-CV). This method considers SAC when partitioning the training and testing subsets. This experiment was conducted on regional landslide susceptibility prediction using different ML models: logistic regression (LR), k-nearest neighbor (KNN), linear discriminant analysis (LDA), artificial neural networks (ANN), support vector machine (SVM), random forest (RF), and C5.0. The experimental results showed that R-CV often produces optimistic performance estimates, e.g., 6–18% higher than those obtained using the S-CV. R-CV also occasionally fails to reveal the true importance of the hyperparameters of models such as SVM and ANN. Additionally, R-CV falsely portrays a considerable improvement in model performance as the number of variables increases. However, this was not the case when the models were evaluated using S-CV. The impact of SAC was more noticeable in complex models such as SVM, RF, and C5.0 (except for ANN) than in simple models such as LDA and LR (except for KNN). Overall, we recommend S-CV over R-CV for a reliable assessment of ML model performance in large-scale LSM. |
| format | Article |
| id | doaj-art-bc4bfcef8be04bc6981e192574e91ef2 |
| institution | Kabale University |
| issn | 2072-4292 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj-art-bc4bfcef8be04bc6981e192574e91ef22025-01-24T13:47:45ZengMDPI AGRemote Sensing2072-42922025-01-0117221310.3390/rs17020213Random Cross-Validation Produces Biased Assessment of Machine Learning Performance in Regional Landslide Susceptibility PredictionChandan Kumar0Gabriel Walton1Paul Santi2Carlos Luza3Department of Geology and Geological Engineering, Colorado School of Mines, Golden, CO 80401, USADepartment of Geology and Geological Engineering, Colorado School of Mines, Golden, CO 80401, USADepartment of Geology and Geological Engineering, Colorado School of Mines, Golden, CO 80401, USADepartment of Geology, Geophysics and Mines, Universidad Nacional de San Agustín, Arequipa 04000, PeruMachine learning (ML) models are extensively used in spatial predictive modeling, including landslide susceptibility prediction. The performance statistics of these models are vital for assessing their reliability, which is typically obtained using the random cross-validation (R-CV) method. However, R-CV has a major drawback, i.e., it ignores the spatial autocorrelation (SAC) inherent in spatial datasets when partitioning the training and testing sets. We assessed the impact of SAC at three crucial phases of ML modeling: hyperparameter tuning, performance evaluation, and learning curve analysis. As an alternative to R-CV, we used spatial cross-validation (S-CV). This method considers SAC when partitioning the training and testing subsets. This experiment was conducted on regional landslide susceptibility prediction using different ML models: logistic regression (LR), k-nearest neighbor (KNN), linear discriminant analysis (LDA), artificial neural networks (ANN), support vector machine (SVM), random forest (RF), and C5.0. The experimental results showed that R-CV often produces optimistic performance estimates, e.g., 6–18% higher than those obtained using the S-CV. R-CV also occasionally fails to reveal the true importance of the hyperparameters of models such as SVM and ANN. Additionally, R-CV falsely portrays a considerable improvement in model performance as the number of variables increases. However, this was not the case when the models were evaluated using S-CV. The impact of SAC was more noticeable in complex models such as SVM, RF, and C5.0 (except for ANN) than in simple models such as LDA and LR (except for KNN). Overall, we recommend S-CV over R-CV for a reliable assessment of ML model performance in large-scale LSM.https://www.mdpi.com/2072-4292/17/2/213landslide susceptibility mappingmachine learningrandom cross-validationspatial autocorrelationspatial cross-validation |
| spellingShingle | Chandan Kumar Gabriel Walton Paul Santi Carlos Luza Random Cross-Validation Produces Biased Assessment of Machine Learning Performance in Regional Landslide Susceptibility Prediction Remote Sensing landslide susceptibility mapping machine learning random cross-validation spatial autocorrelation spatial cross-validation |
| title | Random Cross-Validation Produces Biased Assessment of Machine Learning Performance in Regional Landslide Susceptibility Prediction |
| title_full | Random Cross-Validation Produces Biased Assessment of Machine Learning Performance in Regional Landslide Susceptibility Prediction |
| title_fullStr | Random Cross-Validation Produces Biased Assessment of Machine Learning Performance in Regional Landslide Susceptibility Prediction |
| title_full_unstemmed | Random Cross-Validation Produces Biased Assessment of Machine Learning Performance in Regional Landslide Susceptibility Prediction |
| title_short | Random Cross-Validation Produces Biased Assessment of Machine Learning Performance in Regional Landslide Susceptibility Prediction |
| title_sort | random cross validation produces biased assessment of machine learning performance in regional landslide susceptibility prediction |
| topic | landslide susceptibility mapping machine learning random cross-validation spatial autocorrelation spatial cross-validation |
| url | https://www.mdpi.com/2072-4292/17/2/213 |
| work_keys_str_mv | AT chandankumar randomcrossvalidationproducesbiasedassessmentofmachinelearningperformanceinregionallandslidesusceptibilityprediction AT gabrielwalton randomcrossvalidationproducesbiasedassessmentofmachinelearningperformanceinregionallandslidesusceptibilityprediction AT paulsanti randomcrossvalidationproducesbiasedassessmentofmachinelearningperformanceinregionallandslidesusceptibilityprediction AT carlosluza randomcrossvalidationproducesbiasedassessmentofmachinelearningperformanceinregionallandslidesusceptibilityprediction |