Thermodynamic overfitting and generalization: energetics of predictive intelligence
Overfitting is a crucial concern in machine learning, where an unnecessarily complex model closely captures the details of its training data but fails to generalize to new inputs. Regularization acts as a speed-bump to increasing complexity that ensures models only grow as necessary to fit the funda...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IOP Publishing
2025-01-01
|
| Series: | New Journal of Physics |
| Subjects: | |
| Online Access: | https://doi.org/10.1088/1367-2630/addf71 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Summary: | Overfitting is a crucial concern in machine learning, where an unnecessarily complex model closely captures the details of its training data but fails to generalize to new inputs. Regularization acts as a speed-bump to increasing complexity that ensures models only grow as necessary to fit the fundamental features of data. It mitigates overfitting and improves generalization by imposing a model complexity penalty. We show that both overfitting and regularization are rooted in physics, because they induce an energy penalty in thermodynamic engines that harvest work from data. Overfitting corresponds to divergent energy dissipation, which we identify here for the first time as thermodynamic overfitting . We also introduce thermodynamic regularization , which avoids thermodynamic overfitting by incurring an energy cost from engine complexity. The result is thermodynamic machine learning, which reliably discovers predictive models that generalize to new data by maximizing thermodynamic resources in training. This suggests that the laws of physics jointly create the conditions for emergent complexity and predictive intelligence. These results bridge three distinctive fields: Thermodynamics, Machine Learning, and Computational Mechanics (the information theory of prediction). |
|---|---|
| ISSN: | 1367-2630 |