Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning
While Convolutional Neural Networks (CNNs) excel at learning complex latent-space representations, their over-parameterization can lead to overfitting and reduced performance, particularly with limited data. This, alongside their high computational and memory demands, limits the applicability of CNN...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10851278/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832576772300865536 |
|---|---|
| author | Manish Sharma Jamison Heard Eli Saber Panagiotis Markopoulos |
| author_facet | Manish Sharma Jamison Heard Eli Saber Panagiotis Markopoulos |
| author_sort | Manish Sharma |
| collection | DOAJ |
| description | While Convolutional Neural Networks (CNNs) excel at learning complex latent-space representations, their over-parameterization can lead to overfitting and reduced performance, particularly with limited data. This, alongside their high computational and memory demands, limits the applicability of CNNs for edge deployment and applications where computational resources are constrained. Low-rank matrix approximation has emerged as a promising approach to reduce CNN parameters, but existing methods often require pre-determined ranks or involve complex post-training adjustments, leading to challenges in rank selection, performance loss, and limited practicality in resource-constrained environments. This underscores the need for an adaptive compression method that integrates into the training process, dynamically adjusting model complexity based on data and task requirements. To address this, we propose an efficient training method for CNN compression via dynamic parameter rank pruning. Our approach integrates efficient matrix factorization and novel regularization techniques, forming a robust framework for dynamic rank pruning and model compression. By using Singular Value Decomposition (SVD) to model low-rank convolutional filters and dense weight matrices, and training the SVD factors with back-propagation in an end-to-end manner, we achieve model compression. We evaluate our method on modern CNNs, including ResNet-18, ResNet-20, and ResNet-32, using datasets like CIFAR-10, CIFAR-100, and ImageNet (2012). Our experiments demonstrate that the proposed method can reduce model parameters by up to 50% and improve classification accuracy by up to 2% over baseline models, making CNNs more feasible for practical applications. |
| format | Article |
| id | doaj-art-94387e2af56549a189b7e49471dfad53 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-94387e2af56549a189b7e49471dfad532025-01-31T00:01:24ZengIEEEIEEE Access2169-35362025-01-0113184411845610.1109/ACCESS.2025.353341910851278Convolutional Neural Network Compression via Dynamic Parameter Rank PruningManish Sharma0https://orcid.org/0000-0002-1867-9379Jamison Heard1https://orcid.org/0000-0001-6860-0844Eli Saber2https://orcid.org/0009-0002-8593-4015Panagiotis Markopoulos3https://orcid.org/0000-0001-9686-779XChester F. Carlson Center for Imaging Science, Rochester Institute of Technology, Rochester, NY, USADepartment of Electrical and Microelectronic Engineering, Rochester Institute of Technology, Rochester, NY, USAChester F. Carlson Center for Imaging Science, Rochester Institute of Technology, Rochester, NY, USADepartment of Electrical and Computer Engineering, The University of Texas at San Antonio, San Antonio, TX, USAWhile Convolutional Neural Networks (CNNs) excel at learning complex latent-space representations, their over-parameterization can lead to overfitting and reduced performance, particularly with limited data. This, alongside their high computational and memory demands, limits the applicability of CNNs for edge deployment and applications where computational resources are constrained. Low-rank matrix approximation has emerged as a promising approach to reduce CNN parameters, but existing methods often require pre-determined ranks or involve complex post-training adjustments, leading to challenges in rank selection, performance loss, and limited practicality in resource-constrained environments. This underscores the need for an adaptive compression method that integrates into the training process, dynamically adjusting model complexity based on data and task requirements. To address this, we propose an efficient training method for CNN compression via dynamic parameter rank pruning. Our approach integrates efficient matrix factorization and novel regularization techniques, forming a robust framework for dynamic rank pruning and model compression. By using Singular Value Decomposition (SVD) to model low-rank convolutional filters and dense weight matrices, and training the SVD factors with back-propagation in an end-to-end manner, we achieve model compression. We evaluate our method on modern CNNs, including ResNet-18, ResNet-20, and ResNet-32, using datasets like CIFAR-10, CIFAR-100, and ImageNet (2012). Our experiments demonstrate that the proposed method can reduce model parameters by up to 50% and improve classification accuracy by up to 2% over baseline models, making CNNs more feasible for practical applications.https://ieeexplore.ieee.org/document/10851278/Convolutional neural networkdynamic rank selectionimage classificationlow-rank factorizationmodel compressionmodel pruning |
| spellingShingle | Manish Sharma Jamison Heard Eli Saber Panagiotis Markopoulos Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning IEEE Access Convolutional neural network dynamic rank selection image classification low-rank factorization model compression model pruning |
| title | Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning |
| title_full | Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning |
| title_fullStr | Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning |
| title_full_unstemmed | Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning |
| title_short | Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning |
| title_sort | convolutional neural network compression via dynamic parameter rank pruning |
| topic | Convolutional neural network dynamic rank selection image classification low-rank factorization model compression model pruning |
| url | https://ieeexplore.ieee.org/document/10851278/ |
| work_keys_str_mv | AT manishsharma convolutionalneuralnetworkcompressionviadynamicparameterrankpruning AT jamisonheard convolutionalneuralnetworkcompressionviadynamicparameterrankpruning AT elisaber convolutionalneuralnetworkcompressionviadynamicparameterrankpruning AT panagiotismarkopoulos convolutionalneuralnetworkcompressionviadynamicparameterrankpruning |