A multiobjective continuation method to compute the regularization path of deep neural networks
Sparsity is a highly desired feature in deep neural networks (DNNs) since it ensures numerical efficiency, improves the interpretability (due to the smaller number of relevant features), and robustness. For linear models, it is well known that there exists a regularization path connecting the sparse...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-03-01
|
| Series: | Machine Learning with Applications |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666827025000088 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832540401814208512 |
|---|---|
| author | Augustina Chidinma Amakor Konstantin Sonntag Sebastian Peitz |
| author_facet | Augustina Chidinma Amakor Konstantin Sonntag Sebastian Peitz |
| author_sort | Augustina Chidinma Amakor |
| collection | DOAJ |
| description | Sparsity is a highly desired feature in deep neural networks (DNNs) since it ensures numerical efficiency, improves the interpretability (due to the smaller number of relevant features), and robustness. For linear models, it is well known that there exists a regularization path connecting the sparsest solution in terms of the ℓ1 norm, i.e., zero weights and the non-regularized solution. Recently, there was a first attempt to extend the concept of regularization paths to DNNs by means of treating the empirical loss and sparsity (ℓ1 norm) as two conflicting criteria and solving the resulting multiobjective optimization problem. However, due to the non-smoothness of the ℓ1 norm and the large number of parameters, this approach is not very efficient from a computational perspective. To overcome this limitation, we present an algorithm that allows for the approximation of the entire Pareto front for the above-mentioned objectives in a very efficient manner for high-dimensional DNNs with millions of parameters. We present numerical examples using both deterministic and stochastic gradients. We furthermore demonstrate that knowledge of the regularization path allows for a well-generalizing network parametrization. |
| format | Article |
| id | doaj-art-7f26a7067ab9478aa11823824bfb61a3 |
| institution | Kabale University |
| issn | 2666-8270 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Machine Learning with Applications |
| spelling | doaj-art-7f26a7067ab9478aa11823824bfb61a32025-02-05T04:32:45ZengElsevierMachine Learning with Applications2666-82702025-03-0119100625A multiobjective continuation method to compute the regularization path of deep neural networksAugustina Chidinma Amakor0Konstantin Sonntag1Sebastian Peitz2Department of Computer Science, TU Dortmund University, Joseph-von-Fraunhofer-Straße 25, 44227, Dortmund, Germany; Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany; Corresponding author at: Department of Computer Science, TU Dortmund University, Joseph-von-Fraunhofer-Straße 25, 44227, Dortmund, Germany.Department of Mathematics, Paderborn University, Warburger Str. 100, 33098, Paderborn, GermanyDepartment of Computer Science, TU Dortmund University, Joseph-von-Fraunhofer-Straße 25, 44227, Dortmund, Germany; Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, GermanySparsity is a highly desired feature in deep neural networks (DNNs) since it ensures numerical efficiency, improves the interpretability (due to the smaller number of relevant features), and robustness. For linear models, it is well known that there exists a regularization path connecting the sparsest solution in terms of the ℓ1 norm, i.e., zero weights and the non-regularized solution. Recently, there was a first attempt to extend the concept of regularization paths to DNNs by means of treating the empirical loss and sparsity (ℓ1 norm) as two conflicting criteria and solving the resulting multiobjective optimization problem. However, due to the non-smoothness of the ℓ1 norm and the large number of parameters, this approach is not very efficient from a computational perspective. To overcome this limitation, we present an algorithm that allows for the approximation of the entire Pareto front for the above-mentioned objectives in a very efficient manner for high-dimensional DNNs with millions of parameters. We present numerical examples using both deterministic and stochastic gradients. We furthermore demonstrate that knowledge of the regularization path allows for a well-generalizing network parametrization.http://www.sciencedirect.com/science/article/pii/S2666827025000088Multiobjective optimizationRegularizationContinuation methodNon-smoothHigh-dimensionalProximal gradient |
| spellingShingle | Augustina Chidinma Amakor Konstantin Sonntag Sebastian Peitz A multiobjective continuation method to compute the regularization path of deep neural networks Machine Learning with Applications Multiobjective optimization Regularization Continuation method Non-smooth High-dimensional Proximal gradient |
| title | A multiobjective continuation method to compute the regularization path of deep neural networks |
| title_full | A multiobjective continuation method to compute the regularization path of deep neural networks |
| title_fullStr | A multiobjective continuation method to compute the regularization path of deep neural networks |
| title_full_unstemmed | A multiobjective continuation method to compute the regularization path of deep neural networks |
| title_short | A multiobjective continuation method to compute the regularization path of deep neural networks |
| title_sort | multiobjective continuation method to compute the regularization path of deep neural networks |
| topic | Multiobjective optimization Regularization Continuation method Non-smooth High-dimensional Proximal gradient |
| url | http://www.sciencedirect.com/science/article/pii/S2666827025000088 |
| work_keys_str_mv | AT augustinachidinmaamakor amultiobjectivecontinuationmethodtocomputetheregularizationpathofdeepneuralnetworks AT konstantinsonntag amultiobjectivecontinuationmethodtocomputetheregularizationpathofdeepneuralnetworks AT sebastianpeitz amultiobjectivecontinuationmethodtocomputetheregularizationpathofdeepneuralnetworks AT augustinachidinmaamakor multiobjectivecontinuationmethodtocomputetheregularizationpathofdeepneuralnetworks AT konstantinsonntag multiobjectivecontinuationmethodtocomputetheregularizationpathofdeepneuralnetworks AT sebastianpeitz multiobjectivecontinuationmethodtocomputetheregularizationpathofdeepneuralnetworks |