Physics Department - The Physics behind Optimization and Generalization: A Symmetry-Irreversibility Framework of Learning

Abstract
AI has become an empirical science. We are discovering more and more interesting phenomena in these AI models, but the organizing principles of modern AI have been unclear. In this seminar, I will discuss what I call the Symmetry-Irreversibility Framework (SIF), which leverages symmetry and irreversibility, two main concepts and tools from science in general and physics in particular, to analyze and understand the phenomenology of deep learning. I will discuss how interesting phenomena such as implicit sparsity, collapse, the edge of stability, and the more recently discovered Platonic representation hypothesis could be consequences of the model's hidden symmetries and the irreversibility of the training dynamics. Lastly, I will also discuss how one can apply the SIF to make AI models more efficient, interpretable, and controllable.