Leveraging Simplex Gradient Variance and Bias Reduction for Black-Box Optimization of Noisy and Costly Functions
Gradient variance errors in gradient-based search methods are largely mitigated using momentum, however the bias gradient errors may fail the numerical search methods in reaching the true optimum. We investigate the reduction in both bias and variance errors attributed to the simplex gradient estima...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10843234/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832586880881786880 |
|---|---|
| author | Mircea-Bogdan Radac Titus Nicolae |
| author_facet | Mircea-Bogdan Radac Titus Nicolae |
| author_sort | Mircea-Bogdan Radac |
| collection | DOAJ |
| description | Gradient variance errors in gradient-based search methods are largely mitigated using momentum, however the bias gradient errors may fail the numerical search methods in reaching the true optimum. We investigate the reduction in both bias and variance errors attributed to the simplex gradient estimated from noisy function measurements, in favor of the finite-differences gradient, when both are used for black-box optimization methods. Regardless of the simplex orientation, while reducing the gradient bias error owned to several factors such as truncation, numerical or measurement noise, we claim and verify that, under relaxed assumptions about the underlying function’s differentiability, the estimated gradient by the simplex method has at most half the variance of the finite-difference gradient. The findings are validated with two comprehensive and representative case studies, one related to the minimization of a nonlinear feedback control system cost function and the second related to a deep machine learning classification problem whose hyperparameters are tuned. We conclude that in up to medium-size practical black-box optimization problems with unknown variable domains and where the noisy function measurements are expensive, a simplex gradient-based search is an attractive option. |
| format | Article |
| id | doaj-art-aba0787d35f14259acb177dd3c30cadb |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-aba0787d35f14259acb177dd3c30cadb2025-01-25T00:02:48ZengIEEEIEEE Access2169-35362025-01-0113143041431610.1109/ACCESS.2025.352991510843234Leveraging Simplex Gradient Variance and Bias Reduction for Black-Box Optimization of Noisy and Costly FunctionsMircea-Bogdan Radac0https://orcid.org/0000-0001-8410-6547Titus Nicolae1https://orcid.org/0009-0006-5757-5464Department of Automation and Applied Informatics, Politehnica University of Timisoara, Timisoara, RomaniaSmilecloud, United Business Center 0, Piaţa Consiliul Europei, Timisoara, RomaniaGradient variance errors in gradient-based search methods are largely mitigated using momentum, however the bias gradient errors may fail the numerical search methods in reaching the true optimum. We investigate the reduction in both bias and variance errors attributed to the simplex gradient estimated from noisy function measurements, in favor of the finite-differences gradient, when both are used for black-box optimization methods. Regardless of the simplex orientation, while reducing the gradient bias error owned to several factors such as truncation, numerical or measurement noise, we claim and verify that, under relaxed assumptions about the underlying function’s differentiability, the estimated gradient by the simplex method has at most half the variance of the finite-difference gradient. The findings are validated with two comprehensive and representative case studies, one related to the minimization of a nonlinear feedback control system cost function and the second related to a deep machine learning classification problem whose hyperparameters are tuned. We conclude that in up to medium-size practical black-box optimization problems with unknown variable domains and where the noisy function measurements are expensive, a simplex gradient-based search is an attractive option.https://ieeexplore.ieee.org/document/10843234/Black-boxcontrol systemsdeep learningfinite differencesmachine learningoptimization |
| spellingShingle | Mircea-Bogdan Radac Titus Nicolae Leveraging Simplex Gradient Variance and Bias Reduction for Black-Box Optimization of Noisy and Costly Functions IEEE Access Black-box control systems deep learning finite differences machine learning optimization |
| title | Leveraging Simplex Gradient Variance and Bias Reduction for Black-Box Optimization of Noisy and Costly Functions |
| title_full | Leveraging Simplex Gradient Variance and Bias Reduction for Black-Box Optimization of Noisy and Costly Functions |
| title_fullStr | Leveraging Simplex Gradient Variance and Bias Reduction for Black-Box Optimization of Noisy and Costly Functions |
| title_full_unstemmed | Leveraging Simplex Gradient Variance and Bias Reduction for Black-Box Optimization of Noisy and Costly Functions |
| title_short | Leveraging Simplex Gradient Variance and Bias Reduction for Black-Box Optimization of Noisy and Costly Functions |
| title_sort | leveraging simplex gradient variance and bias reduction for black box optimization of noisy and costly functions |
| topic | Black-box control systems deep learning finite differences machine learning optimization |
| url | https://ieeexplore.ieee.org/document/10843234/ |
| work_keys_str_mv | AT mirceabogdanradac leveragingsimplexgradientvarianceandbiasreductionforblackboxoptimizationofnoisyandcostlyfunctions AT titusnicolae leveragingsimplexgradientvarianceandbiasreductionforblackboxoptimizationofnoisyandcostlyfunctions |