Department of Industrial Engineering & decision Analytics [Joint IEDA/ISOM] seminar - Taming "data-hungry" reinforcement learning? Stability in continuous state-action spaces

We introduce a novel framework for analyzing reinforcement learning (RL) in continuous state-action spaces and use it to prove fast rates of convergence in both off-line and on-line settings. Our analysis highlights two key stability properties that relate to how changes in value functions and/or policies affect the Bellman operator and occupation measures. We argue that these properties are satisfied in many continuous state-action Markov decision processes. In this talk, we illustrate the intuition behind these properties through a simple inventory management example. Our framework also sheds new light on the roles of pessimism and optimism in RL. Contrary to earlier beliefs about their necessity for managing uncertainty, our work shows that while these approaches can be useful for obtaining a pilot estimate, they are not essential in later stages of training.