Advances in Off-policy Value Estimation in Reinforcement Learning

Temporal difference learning algorithms underlie most approaches in reinforcement learning, for both prediction and control. A well-known issue is that these approaches can diverge under nonlinear function approximation, such as with neural networks, and in the off-policy setting where data is generated by a different policy than the one being learned. Naturally, there has been a flurry of work towards resolving this issue. In this talk, I will discuss two key advances that largely resolve the problem: sound gradient-based methods and emphatic reweightings. I will discuss our generalized objective that unifies several approaches and facilitates creating easy-to-use algorithms that consistently outperform temporal difference learning approaches in our experiments.