ECE Seminar - LQR learning pipelines: between reinforcement learning and adaptive control

Abstract: Over the past decades, adaptive control and reinforcement learning (RL) have developed largely in parallel, despite sharing deep roots in optimal control. While adaptive control emphasizes robust stability under uncertainty, RL focuses on optimality through learning from data. Today, these fields are increasingly converging, opening the door to new data-driven control pipelines that combine the best of both worlds. At the heart of this convergence lies the LQR problem, a benchmark for both optimal and adaptive control. Despite its rich history, fundamental gaps remain: we lack methods that are simultaneously direct (model-free), adaptive (online), cheaply implementable, and with closed-loop stability and optimality certificates.

In this talk, we introduce LQR learning pipelines with policy gradients that touches all of the above merits. Our method adaptively updates the policy in feedback by descending the (in)direct gradient of the LQR cost, which can be explicitly computed using a batch of persistently exciting data. We showcase the merits of using natural gradient and Gauss-Newton methods for faster and more robust policy updates, and we propose a regularization method to further improve performance and robust stability. We highlight theoretical certificates for closed-loop stability, optimality, and robustness. Finally, we demonstrate the computational and sample efficiency of our methods via simulations and real-world applications.