AReLU: Agile Rectified Linear Unit for Improving Lightweight Convolutional Neural Networks
Dynamic activation functions usually gain remarkable improvements for neural networks. Dynamic activation functions depending on input features show better performance than the input-independents. But the improvements are achieved with extra memory and computational cost, which is non-negligible for...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10843665/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832586894779613184 |
|---|---|
| author | Fu Chen Yepeng Guan |
| author_facet | Fu Chen Yepeng Guan |
| author_sort | Fu Chen |
| collection | DOAJ |
| description | Dynamic activation functions usually gain remarkable improvements for neural networks. Dynamic activation functions depending on input features show better performance than the input-independents. But the improvements are achieved with extra memory and computational cost, which is non-negligible for lightweight convolutional networks. To address this issue, a lightweight input-dependent dynamic activation function is proposed, namely, Agile Rectified Linear Unit (AReLU). And Parallel Local Cross-Feature Interaction (PLCFI) is proposed as the lightweight feature extractor of AReLU. Considering the multi-channel characteristic of input features, Channel-Exclusive AReLU (CE-AReLU) and Channel-Shared AReLU (CS-AReLU) are designed based on PLCFI. CE-AReLU learns exclusive structure of activation function for each channel, while the functional structure of CS-AReLU is shared by channels. Furthermore, a specific initialization method for neurons in PLCFI is proposed to facilitate model convergence, called imitative initialization. Extensive experiments on different datasets and lightweight convolutional neural network architectures demonstrate the superiority and generality of AReLU. It shows that CE-AReLU gains remarkable performance in common visual classification benchmarks. Moreover, CS-AReLU shows its capacity of capturing dependencies among different samples in a batch, achieving higher recognition accuracy on fine-grained classification task. |
| format | Article |
| id | doaj-art-e1ae8d899475438b92e838b393e98490 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-e1ae8d899475438b92e838b393e984902025-01-25T00:01:20ZengIEEEIEEE Access2169-35362025-01-0113143791439110.1109/ACCESS.2025.353042410843665AReLU: Agile Rectified Linear Unit for Improving Lightweight Convolutional Neural NetworksFu Chen0https://orcid.org/0009-0005-4646-2491Yepeng Guan1https://orcid.org/0000-0003-1854-1430School of Communication and Information Engineering, Shanghai University, Shanghai, ChinaSchool of Communication and Information Engineering, Shanghai University, Shanghai, ChinaDynamic activation functions usually gain remarkable improvements for neural networks. Dynamic activation functions depending on input features show better performance than the input-independents. But the improvements are achieved with extra memory and computational cost, which is non-negligible for lightweight convolutional networks. To address this issue, a lightweight input-dependent dynamic activation function is proposed, namely, Agile Rectified Linear Unit (AReLU). And Parallel Local Cross-Feature Interaction (PLCFI) is proposed as the lightweight feature extractor of AReLU. Considering the multi-channel characteristic of input features, Channel-Exclusive AReLU (CE-AReLU) and Channel-Shared AReLU (CS-AReLU) are designed based on PLCFI. CE-AReLU learns exclusive structure of activation function for each channel, while the functional structure of CS-AReLU is shared by channels. Furthermore, a specific initialization method for neurons in PLCFI is proposed to facilitate model convergence, called imitative initialization. Extensive experiments on different datasets and lightweight convolutional neural network architectures demonstrate the superiority and generality of AReLU. It shows that CE-AReLU gains remarkable performance in common visual classification benchmarks. Moreover, CS-AReLU shows its capacity of capturing dependencies among different samples in a batch, achieving higher recognition accuracy on fine-grained classification task.https://ieeexplore.ieee.org/document/10843665/Neural networkdynamic activation functionlearnable activation functionlightweight CNN |
| spellingShingle | Fu Chen Yepeng Guan AReLU: Agile Rectified Linear Unit for Improving Lightweight Convolutional Neural Networks IEEE Access Neural network dynamic activation function learnable activation function lightweight CNN |
| title | AReLU: Agile Rectified Linear Unit for Improving Lightweight Convolutional Neural Networks |
| title_full | AReLU: Agile Rectified Linear Unit for Improving Lightweight Convolutional Neural Networks |
| title_fullStr | AReLU: Agile Rectified Linear Unit for Improving Lightweight Convolutional Neural Networks |
| title_full_unstemmed | AReLU: Agile Rectified Linear Unit for Improving Lightweight Convolutional Neural Networks |
| title_short | AReLU: Agile Rectified Linear Unit for Improving Lightweight Convolutional Neural Networks |
| title_sort | arelu agile rectified linear unit for improving lightweight convolutional neural networks |
| topic | Neural network dynamic activation function learnable activation function lightweight CNN |
| url | https://ieeexplore.ieee.org/document/10843665/ |
| work_keys_str_mv | AT fuchen areluagilerectifiedlinearunitforimprovinglightweightconvolutionalneuralnetworks AT yepengguan areluagilerectifiedlinearunitforimprovinglightweightconvolutionalneuralnetworks |