A dynamic dropout self-distillation method for object segmentation
Abstract There is a phenomenon that better teachers cannot teach out better students in knowledge distillation due to the capacity mismatch. Especially in pixel-level object segmentation, there are some challenging pixels that are difficult for the student model to learn. Even if the student model l...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Springer
2024-12-01
|
| Series: | Complex & Intelligent Systems |
| Subjects: | |
| Online Access: | https://doi.org/10.1007/s40747-024-01705-8 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832571205987598336 |
|---|---|
| author | Lei Chen Tieyong Cao Yunfei Zheng Yang Wang Bo Zhang Jibin Yang |
| author_facet | Lei Chen Tieyong Cao Yunfei Zheng Yang Wang Bo Zhang Jibin Yang |
| author_sort | Lei Chen |
| collection | DOAJ |
| description | Abstract There is a phenomenon that better teachers cannot teach out better students in knowledge distillation due to the capacity mismatch. Especially in pixel-level object segmentation, there are some challenging pixels that are difficult for the student model to learn. Even if the student model learns from the teacher model for each pixel, the student’s performance still struggles to show significant improvement. Mimicking the learning process of human beings from easy to difficult, a dynamic dropout self-distillation method for object segmentation is proposed, which solves this problem by discarding the knowledge that the student struggles to learn. Firstly, the pixels where there is a significant difference between the teacher and student models are found according to the predicted probabilities. And these pixels are defined as difficult-to-learn pixel for the student model. Secondly, a dynamic dropout strategy is proposed to match the capability variation of the student model, which is used to discard the pixels with hard knowledge for the student model. Finally, to validate the effectiveness of the proposed method, a simple student model for object segmentation and a virtual teacher model with perfect segmentation accuracy are constructed. Experiment results on four public datasets demonstrate that, when there is a large performance gap between the teacher and student models, the proposed self-distillation method is more effective in improving the performance of the student model compared to other methods. |
| format | Article |
| id | doaj-art-ed807f233422450ea8bf142bbc5a88b5 |
| institution | Kabale University |
| issn | 2199-4536 2198-6053 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | Springer |
| record_format | Article |
| series | Complex & Intelligent Systems |
| spelling | doaj-art-ed807f233422450ea8bf142bbc5a88b52025-02-02T12:50:20ZengSpringerComplex & Intelligent Systems2199-45362198-60532024-12-0111111410.1007/s40747-024-01705-8A dynamic dropout self-distillation method for object segmentationLei Chen0Tieyong Cao1Yunfei Zheng2Yang Wang3Bo Zhang4Jibin Yang5The Army Engineering University of PLAThe Army Engineering University of PLAThe Army Engineering University of PLAThe Army Engineering University of PLAInstitute of International Relations, National Defense University of Science and TechnologyThe Army Engineering University of PLAAbstract There is a phenomenon that better teachers cannot teach out better students in knowledge distillation due to the capacity mismatch. Especially in pixel-level object segmentation, there are some challenging pixels that are difficult for the student model to learn. Even if the student model learns from the teacher model for each pixel, the student’s performance still struggles to show significant improvement. Mimicking the learning process of human beings from easy to difficult, a dynamic dropout self-distillation method for object segmentation is proposed, which solves this problem by discarding the knowledge that the student struggles to learn. Firstly, the pixels where there is a significant difference between the teacher and student models are found according to the predicted probabilities. And these pixels are defined as difficult-to-learn pixel for the student model. Secondly, a dynamic dropout strategy is proposed to match the capability variation of the student model, which is used to discard the pixels with hard knowledge for the student model. Finally, to validate the effectiveness of the proposed method, a simple student model for object segmentation and a virtual teacher model with perfect segmentation accuracy are constructed. Experiment results on four public datasets demonstrate that, when there is a large performance gap between the teacher and student models, the proposed self-distillation method is more effective in improving the performance of the student model compared to other methods.https://doi.org/10.1007/s40747-024-01705-8Self-distillationObject segmentationDynamic dropoutCapacity mismatch |
| spellingShingle | Lei Chen Tieyong Cao Yunfei Zheng Yang Wang Bo Zhang Jibin Yang A dynamic dropout self-distillation method for object segmentation Complex & Intelligent Systems Self-distillation Object segmentation Dynamic dropout Capacity mismatch |
| title | A dynamic dropout self-distillation method for object segmentation |
| title_full | A dynamic dropout self-distillation method for object segmentation |
| title_fullStr | A dynamic dropout self-distillation method for object segmentation |
| title_full_unstemmed | A dynamic dropout self-distillation method for object segmentation |
| title_short | A dynamic dropout self-distillation method for object segmentation |
| title_sort | dynamic dropout self distillation method for object segmentation |
| topic | Self-distillation Object segmentation Dynamic dropout Capacity mismatch |
| url | https://doi.org/10.1007/s40747-024-01705-8 |
| work_keys_str_mv | AT leichen adynamicdropoutselfdistillationmethodforobjectsegmentation AT tieyongcao adynamicdropoutselfdistillationmethodforobjectsegmentation AT yunfeizheng adynamicdropoutselfdistillationmethodforobjectsegmentation AT yangwang adynamicdropoutselfdistillationmethodforobjectsegmentation AT bozhang adynamicdropoutselfdistillationmethodforobjectsegmentation AT jibinyang adynamicdropoutselfdistillationmethodforobjectsegmentation AT leichen dynamicdropoutselfdistillationmethodforobjectsegmentation AT tieyongcao dynamicdropoutselfdistillationmethodforobjectsegmentation AT yunfeizheng dynamicdropoutselfdistillationmethodforobjectsegmentation AT yangwang dynamicdropoutselfdistillationmethodforobjectsegmentation AT bozhang dynamicdropoutselfdistillationmethodforobjectsegmentation AT jibinyang dynamicdropoutselfdistillationmethodforobjectsegmentation |