FINTECH THRUST SEMINAR | Optimization and Learning for Mean-Field Games via Occupation Measure
Optimization and Learning for Mean-Field Games via Occupation Measure
Abstract:
Understanding how large groups of agents interact is critical in fields like economics, finance, and engineering. However, directly analyzing $N$-player games becomes infeasible as the number of agents grows. Mean-field games (MFGs) provide a powerful solution by approximating these complex systems, allowing us to study the collective behavior of a large population as the limit of the $N$-player game when $N$ is very large. This approximation significantly simplifies analysis while retaining key insights about the system.
In this talk, we present recent advancements at the intersection of MFGs and multi-agent reinforcement learning (MARL). First, we introduce Mean-Field Occupation Measure Optimization (MF-OMO), a novel framework that reinterprets Nash equilibria in discrete-time MFGs as a single optimization problem over occupation measures. This reformulation avoids restrictive assumptions often required by traditional methods and enables the use of standard optimization algorithms to find multiple equilibria. We also extend this approach to continuous-time finite-state MFGs, broadening its applicability.
Building on the concept of occupation measures, we then propose Mean-Field Occupation Measure Learning (MF-OML),the first fully polynomial online reinforcement learning algorithm capable of finding approximate Nash equilibria in large population games beyond zero-sum and potential games. We establish regret bounds for the $N$-player games that can be approximated by MFGs under monotonicity conditions.
Together, these advancements provide a comprehensive approach to characterizing and solving Nash equilibria in complex multi-agent environments.
The talk is based on two papers:
[1] Guo, X., Hu, A., & Zhang, J. (2024). MF-OMO: An optimization formulation of mean-field games. SIAM Journal on Control and Optimization, 62(1), 243-270.
[2] Hu, A., & Zhang, J. (2024). MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games. arXiv preprint arXiv:2405.00282.
Anran Hu works at the intersection of stochastic control, game theory, optimization and machine learning. Her primary research areas are mean-field games, continuous-time stochastic control and reinforcement learning. She is also interested in FinTech and applying machine learning and reinforcement learning to finance.
A main focus of Hu's research is on the reinforcement learning of large-population games and continuous-time stochastic control systems. Another important thread of Hu's work is on developing optimization formulations and novel modeling generalizations of mean-field games and their extensions. Her research spans the full spectrum from fundamentally new frameworks, analytical theory, efficient algorithms to open-source software for such important problems which have wide applications in finance, economy, transportation, robotics and energy systems.
Before coming to Columbia University, Hu was a Hooke Research Fellow in the Mathematical Institute at the University of Oxford. She completed her Ph.D. in Berkeley IEOR, advised by Prof. Xin Guo. Before coming to Berkeley, she obtained my B.S. degree from the School of Mathematical Sciences, Peking University.