Advancing privacy-aware machine learning on sensitive data via edge-based continual µ-training for personalized large models
This paper introduces an innovative method for fine-tuning a large multi-label model for abnormality detection, utilizing a smaller trainer and advanced knowledge distillation techniques. It studies the effects of fine-tuning on various abnormalities, noting different improvements based on the Origi...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IOP Publishing
2025-01-01
|
| Series: | Machine Learning: Science and Technology |
| Subjects: | |
| Online Access: | https://doi.org/10.1088/2632-2153/adaca3 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832575977005252608 |
|---|---|
| author | Zhaojing Huang Leping Yu Luis Fernando Herbozo Contreras Kamran Eshraghian Nhan Duy Truong Armin Nikpour Omid Kavehei |
| author_facet | Zhaojing Huang Leping Yu Luis Fernando Herbozo Contreras Kamran Eshraghian Nhan Duy Truong Armin Nikpour Omid Kavehei |
| author_sort | Zhaojing Huang |
| collection | DOAJ |
| description | This paper introduces an innovative method for fine-tuning a large multi-label model for abnormality detection, utilizing a smaller trainer and advanced knowledge distillation techniques. It studies the effects of fine-tuning on various abnormalities, noting different improvements based on the Original Model’s performance in specific tasks. The experimental setup, optimized for on-device inference and fine-tuning with limited computational resources, demonstrates moderate yet promising enhancements in model performance post-fine-tuning. Key insights from the study include the significance of aligning the µ -Trainer’s behavior with the Original Model and the influence of hyper-parameters like batch size on fine-tuning outcomes. The research acknowledges limitations such as the limited exploration of loss functions in multi-label models and constraints in architectural design, suggesting potential avenues for future investigation. While the proposed Naive Continual Fine-tuning Process is in its early stages, we highlight this paper’s potential model personalization on long-term data. Moreover, weight transfer in our system is exclusively for fine-tuning; hence, it improves user privacy protection by failing data reconstruction attempts from weights, like an issue with Federated learning models. Our on-device fine-tuning prevents the transferring of data or gradients from the edge of the network to their server. Despite modest performance improvements after fine-tuning, these working layers represent a small fraction (0.7%) of the total weights in the Original Model and 1.6% in the µ -Trainer. This study establishes a foundational framework for advancing personalized model adaptation, on-device inference and fine-tuning while emphasizing the importance of safeguarding data privacy in model development. |
| format | Article |
| id | doaj-art-12e387a7bc5540d2b88984749ee7e38a |
| institution | Kabale University |
| issn | 2632-2153 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IOP Publishing |
| record_format | Article |
| series | Machine Learning: Science and Technology |
| spelling | doaj-art-12e387a7bc5540d2b88984749ee7e38a2025-01-31T13:36:38ZengIOP PublishingMachine Learning: Science and Technology2632-21532025-01-016101502510.1088/2632-2153/adaca3Advancing privacy-aware machine learning on sensitive data via edge-based continual µ-training for personalized large modelsZhaojing Huang0https://orcid.org/0009-0004-2796-6734Leping Yu1https://orcid.org/0009-0008-4794-9586Luis Fernando Herbozo Contreras2https://orcid.org/0009-0001-8458-9486Kamran Eshraghian3https://orcid.org/0009-0007-5372-8010Nhan Duy Truong4https://orcid.org/0000-0003-4350-8026Armin Nikpour5https://orcid.org/0000-0002-2384-0710Omid Kavehei6https://orcid.org/0000-0002-2753-5553School of Biomedical Engineering, The University of Sydney , Darlington, NSW 2008, AustraliaSchool of Biomedical Engineering, The University of Sydney , Darlington, NSW 2008, AustraliaSchool of Biomedical Engineering, The University of Sydney , Darlington, NSW 2008, AustraliaiDataMap Corporation , Perth, WA 6149, AustraliaSchool of Biomedical Engineering, The University of Sydney , Darlington, NSW 2008, AustraliaDepartment of Neurology, Royal Prince Alfred Hospital, and Central Clinical School, The University of Sydney , Camperdown, NSW 2006, AustraliaSchool of Biomedical Engineering, The University of Sydney , Darlington, NSW 2008, AustraliaThis paper introduces an innovative method for fine-tuning a large multi-label model for abnormality detection, utilizing a smaller trainer and advanced knowledge distillation techniques. It studies the effects of fine-tuning on various abnormalities, noting different improvements based on the Original Model’s performance in specific tasks. The experimental setup, optimized for on-device inference and fine-tuning with limited computational resources, demonstrates moderate yet promising enhancements in model performance post-fine-tuning. Key insights from the study include the significance of aligning the µ -Trainer’s behavior with the Original Model and the influence of hyper-parameters like batch size on fine-tuning outcomes. The research acknowledges limitations such as the limited exploration of loss functions in multi-label models and constraints in architectural design, suggesting potential avenues for future investigation. While the proposed Naive Continual Fine-tuning Process is in its early stages, we highlight this paper’s potential model personalization on long-term data. Moreover, weight transfer in our system is exclusively for fine-tuning; hence, it improves user privacy protection by failing data reconstruction attempts from weights, like an issue with Federated learning models. Our on-device fine-tuning prevents the transferring of data or gradients from the edge of the network to their server. Despite modest performance improvements after fine-tuning, these working layers represent a small fraction (0.7%) of the total weights in the Original Model and 1.6% in the µ -Trainer. This study establishes a foundational framework for advancing personalized model adaptation, on-device inference and fine-tuning while emphasizing the importance of safeguarding data privacy in model development.https://doi.org/10.1088/2632-2153/adaca3knowledge distillationedged devicefine-tuningtransfer learningheart abnormalitiescardiovascular diseases |
| spellingShingle | Zhaojing Huang Leping Yu Luis Fernando Herbozo Contreras Kamran Eshraghian Nhan Duy Truong Armin Nikpour Omid Kavehei Advancing privacy-aware machine learning on sensitive data via edge-based continual µ-training for personalized large models Machine Learning: Science and Technology knowledge distillation edged device fine-tuning transfer learning heart abnormalities cardiovascular diseases |
| title | Advancing privacy-aware machine learning on sensitive data via edge-based continual µ-training for personalized large models |
| title_full | Advancing privacy-aware machine learning on sensitive data via edge-based continual µ-training for personalized large models |
| title_fullStr | Advancing privacy-aware machine learning on sensitive data via edge-based continual µ-training for personalized large models |
| title_full_unstemmed | Advancing privacy-aware machine learning on sensitive data via edge-based continual µ-training for personalized large models |
| title_short | Advancing privacy-aware machine learning on sensitive data via edge-based continual µ-training for personalized large models |
| title_sort | advancing privacy aware machine learning on sensitive data via edge based continual µ training for personalized large models |
| topic | knowledge distillation edged device fine-tuning transfer learning heart abnormalities cardiovascular diseases |
| url | https://doi.org/10.1088/2632-2153/adaca3 |
| work_keys_str_mv | AT zhaojinghuang advancingprivacyawaremachinelearningonsensitivedataviaedgebasedcontinualμtrainingforpersonalizedlargemodels AT lepingyu advancingprivacyawaremachinelearningonsensitivedataviaedgebasedcontinualμtrainingforpersonalizedlargemodels AT luisfernandoherbozocontreras advancingprivacyawaremachinelearningonsensitivedataviaedgebasedcontinualμtrainingforpersonalizedlargemodels AT kamraneshraghian advancingprivacyawaremachinelearningonsensitivedataviaedgebasedcontinualμtrainingforpersonalizedlargemodels AT nhanduytruong advancingprivacyawaremachinelearningonsensitivedataviaedgebasedcontinualμtrainingforpersonalizedlargemodels AT arminnikpour advancingprivacyawaremachinelearningonsensitivedataviaedgebasedcontinualμtrainingforpersonalizedlargemodels AT omidkavehei advancingprivacyawaremachinelearningonsensitivedataviaedgebasedcontinualμtrainingforpersonalizedlargemodels |