DSA & IoT Joint Seminar | Structured Reinforcement Learning for Intelligent Decision-Making Networked Systems: Foundations and Practice

A wide variety of real-world applications in next-generation intelligent decision-making networked systems, such as edge/cloud computing, autonomous driving and healthcare, involve complex resource-constrained sequential decision-making problems. Reinforcement learning (RL) has recently proven its worth in these domains and is beginning to show some success. However, model-based approaches are prone to system inaccuracies, and model-free methods suffer from the curse of dimensionality. These issues are further exacerbated when multiple coupled Markov Decision Processes (MDPs) are involved. To address these challenges, we advocate a Structured RL framework that leverages inherent problem structure encoded in these applications to design new learning architectures for improved sample efficiency and accelerated learning speed. Specifically, we model those resource-constrained sequential decision-making problems as a restless multi-armed bandits (RMAB) problem, which is provably computationally intractable. The unknown dynamics of those coupled MDPs in the RMAB further present significant hurdles for existing offline, index-based algorithms. To bridge this gap, we first propose novel low-complexity index policies to address dimensionality concerns, which are provably optimal. We then leverage these indices to develop index-aware RL algorithms to manage the exploration-exploitation tradeoff with provably sublinear learning regret guarantees. Finally, we demonstrate one practical application of our proposed structured RL framework in real-world domains, i.e., adaptive multi-user video streaming in the wireless edge, which underscores the significant potentials of structured RL in addressing complex and dynamic resource-constrained sequential decision-making problems.