LIDS Tea Talk | Kai Yun & Kaveh Alimohammadi

Kai Yun
"Provably Safe, Yet Scalable Reinforcement Learning"

Safe reinforcement learning (RL) aims to learn policies that optimize rewards while satisfying constraints. Predominant approaches rely on soft-constrained policy optimization, which has achieved empirical success but does not provide formal safety guarantees for the learned policy. In contrast, methods with strict guarantees typically rely on explicit certificate functions, whose construction requires the direct synthesis and verification of control-invariant sets, a process that scales poorly with state dimension and often yields overly conservative behavior.

In this paper, we present the Provably Safe, yet Scalable RL (PS2-RL) framework, a novel two-phase architecture for learning provably safe policies in a scalable manner, designed to overcome the key bottlenecks of prior methods. Rather than explicitly computing invariant sets, PS2-RL leverages a learned backup policy to forward-integrate the system dynamics, generating an implicit control-invariant set online. In the first phase, the backup policy is trained with our proposed safe-arrival value function, which characterizes the optimal backup policy for invariant-set construction. In the second phase, an RL policy is trained end-to-end through a differentiable projection layer that strictly enforces the safety guarantees induced by the learned backup policy.

By maximizing the volume of the implicit control-invariant set in the first phase, the resulting PS2 policy from the second phase is performant and scalable, while maintaining provable safety. Crucially, PS2-RL imposes no restrictions on the underlying RL algorithm and can be plugged into any existing training pipeline. We establish theoretical guarantees for the proposed framework and evaluate it on robotic control tasks with state dimensions up to 10, a regime in which prior provably safe RL methods struggle or become impractical.

Kai is a 2nd-year PhD student working with Navid Azizan. His research focuses on safe robotics and trustworthy AI, bridging formal control theory with machine learning to ensure the reliable deployment of autonomous systems.

‍

Kaveh Alimohammadi
"Retrieval and Reasoning in LLM Reranking"

Large language models (LLMs) are increasingly used to rerank recommendations using both step-by-step reasoning and retrieved evidence, such as customer reviews. We show that simply retrieving a few relevant reviews matches the accuracy of long reasoning traces at a fraction of the computational cost, and combining both signals yields the best performance.

However, we uncover a critical vulnerability: LLMs suffer from an "availability bias," systematically favoring items simply because they have attached text; even when that text is unhelpful, irrelevant, or assigned to the wrong item. By introducing controlled benchmarks to isolate this bias, we prove that genuine evidence quality boosts recommendation accuracy whereas naive evaluations often mistake mere text placement for actual information value.

Ultimately, while retrieved evidence is a powerful asset for LLM rerankers, trustworthy evaluation requires controlling for how evidence is presented and attributed.

Kaveh is a fourth-year Ph.D. student in IDSS at MIT and a student member of LIDS. His research focuses on the post-training of large language models, with an emphasis on inference-time scaling, uncertainty quantification, and efficient inference algorithms.

‍

ABOUT LIDS TEA TALKS:

Tea talks are 20-minute-long informal chalk-talks for the purpose of sharing ideas and creating awareness about some of the topics of interest to the LIDS audience, followed by a Q&A.
Lunch is provided.