Advancing privacy-aware machine learning on sensitive data via edge-based continual µ-training for personalized large models
This paper introduces an innovative method for fine-tuning a large multi-label model for abnormality detection, utilizing a smaller trainer and advanced knowledge distillation techniques. It studies the effects of fine-tuning on various abnormalities, noting different improvements based on the Origi...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IOP Publishing
2025-01-01
|
| Series: | Machine Learning: Science and Technology |
| Subjects: | |
| Online Access: | https://doi.org/10.1088/2632-2153/adaca3 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Summary: | This paper introduces an innovative method for fine-tuning a large multi-label model for abnormality detection, utilizing a smaller trainer and advanced knowledge distillation techniques. It studies the effects of fine-tuning on various abnormalities, noting different improvements based on the Original Model’s performance in specific tasks. The experimental setup, optimized for on-device inference and fine-tuning with limited computational resources, demonstrates moderate yet promising enhancements in model performance post-fine-tuning. Key insights from the study include the significance of aligning the µ -Trainer’s behavior with the Original Model and the influence of hyper-parameters like batch size on fine-tuning outcomes. The research acknowledges limitations such as the limited exploration of loss functions in multi-label models and constraints in architectural design, suggesting potential avenues for future investigation. While the proposed Naive Continual Fine-tuning Process is in its early stages, we highlight this paper’s potential model personalization on long-term data. Moreover, weight transfer in our system is exclusively for fine-tuning; hence, it improves user privacy protection by failing data reconstruction attempts from weights, like an issue with Federated learning models. Our on-device fine-tuning prevents the transferring of data or gradients from the edge of the network to their server. Despite modest performance improvements after fine-tuning, these working layers represent a small fraction (0.7%) of the total weights in the Original Model and 1.6% in the µ -Trainer. This study establishes a foundational framework for advancing personalized model adaptation, on-device inference and fine-tuning while emphasizing the importance of safeguarding data privacy in model development. |
|---|---|
| ISSN: | 2632-2153 |