Safe and Efficient Exploration in Reinforcement Learning

Monday, November 23, 2020 - 11:00am to 12:00pm

Event Calendar Category

LIDS Seminar Series

Speaker Name

Andreas Krause

Affiliation

ETH Zürich

Zoom meeting id

995 6944 6655

Join Zoom meeting

https://mit.zoom.us/j/99569446655

Abstract

At the heart of Reinforcement Learning lies the challenge of trading exploration -- collecting data for identifying better models -- and exploitation -- using the estimate to make decisions.  In simulated environments (e.g., games), exploration is primarily a computational challenge.  In real-world settings, exploration is costly, and a potentially dangerous proposition, as it requires experimenting with actions that have unknown consequences.  In this talk, I will present our work towards rigorously reasoning about the safety of exploration in reinforcement learning.  I will discuss a model-free approach, where we seek to optimize an unknown reward function subject to unknown constraints.  Both reward and constraints are revealed through noisy experiments, and safety requires that no infeasible action is chosen at any time. I will also discuss model-based approaches, where we learn about system dynamics through exploration, yet need to verify the safety of the estimated policy.  Our approaches use Bayesian inference over the objective, constraints, and dynamics, and -- under some regularity conditions -- are guaranteed to be both safe and complete, i.e., converge to a natural notion of reachable optimum.  I will also present recent results harnessing the model uncertainty for improving the efficiency of exploration, and show experiments on safely and efficiently tuning cyber-physical systems in a data-driven manner.

Biography

Andreas Krause is a Professor of Computer Science at ETH Zurich, where he leads the Learning & Adaptive Systems Group. He also serves as Academic Co-Director of the Swiss Data Science Center and Chair of the ETH AI Center. Before that, he was an Assistant Professor of Computer Science at Caltech. He received his Ph.D. in Computer Science from Carnegie Mellon University (2008) and his Diplom in Computer Science and Mathematics from the Technical University of Munich, Germany (2004). He is a Microsoft Research Faculty Fellow and a Kavli Frontiers Fellow of the US National Academy of Sciences. He received ERC Starting Investigator and ERC Consolidator grants, the Deutscher Mustererkennungspreis, an NSF CAREER award as well as the ETH Golden Owl teaching award. His research has received awards at several premier conferences and journals, including the ACM SIGKDD Test of Time award 2019 and the ICML Test of Time award 2020. Andreas Krause served as Program Co-Chair for ICML 2018 and he is serving as Action Editor for the Journal of Machine Learning Research.