Thermodynamic overfitting and generalization: energetics of predictive intelligence - Alexander B Boyd; James P Crutchfield; Mile Gu; Felix C Binder (New Journal of Physics)
Overfitting is a crucial concern in machine learning, where an unnecessarily complex model closely captures the details of its training data but fails to generalize to new inputs. Regularization acts as a speed-bump to increasing complexity that ensures models only grow as necessary to fit the fundamental features of data. It mitigates overfitting and improves generalization by imposing a model complexity penalty. We show that both overfitting and regularization are rooted in physics, because they induce an energy penalty in thermodynamic engines that harvest work from data. Overfitting corresponds to divergent energy dissipation, which we identify here for the first time as thermodynamic overfitting . We also introduce thermodynamic regularization , which avoids thermodynamic overfitting by incurring an energy cost from engine complexity. The result is thermodynamic machine learning, which reliably discovers predictive models that generalize to new data by maximizing thermodynamic resources in training. This suggests that the laws of physics jointly create the conditions for emergent complexity and predictive intelligence. These results bridge three distinctive fields: Thermodynamics, Machine Learning, and Computational Mechanics (the information theory of prediction).