back to list

Project: Reinforcement Learning under Dynamic Safety Bounds

Description

Safe Reinforcement Learning (Safe RL) typically assumes a fixed safety bound or cost threshold, constraining the agent’s behavior during training and evaluation. However, in many real-world applications (e.g., robotics, autonomous driving, or healthcare), safety requirements are not static. Regulatory limits, environmental constraints, or critical tolerances may shift over time, sometimes becoming stricter, other times relaxing.

This project will introduce a continual learning setting for Safe RL, where the safety bound (cost threshold) changes periodically or in unknown patterns across tasks. The agent must continually adapt to stricter or looser safety requirements while retaining competence across all previously encountered bounds. This presents a dual challenge: avoiding catastrophic forgetting (as in continual learning) and satisfying evolving safety constraints (as in Safe RL).

Objectives:

  1. Design evaluation scenarios
    • Define task sequences where the safety bound/threshold dynamically changes (e.g., periodically increasing/decreasing, randomly fluctuating, or shifting unpredictably).
    • Develop multiple settings (deterministic vs. stochastic changes, known vs. unknown transitions).
  2. Implement baseline algorithms
    • Safe RL methods (e.g., Constrained Policy Optimization, Lagrangian methods, Saute RL).
    • Continual learning adaptations (e.g., EWC, MAS, AGEM) for safe settings.
  3. Evaluate continual adaptation
    • Assess how methods adapt to new safety requirements while retaining performance on old ones.
    • Measure trade-offs between safety, reward optimization, and memory retention.
  4. Analyze failure modes
    • Investigate cases if and where agents forget how to act under looser bounds after adapting to stricter ones (and vice versa).
    • Identify bottlenecks in scaling to many safety bound changes.

Outcomes:

  • A benchmarking framework for continual Safe RL with dynamic safety constraints.
  • Evaluation of Safe RL algorithms extended to continual settings.
  • Metrics and analysis tools for measuring adaptation, forgetting, and safety compliance across tasks.
  • Insights into how continual learning techniques can be integrated with Safe RL.
  • A novel method that outperforms existing baselines in this setting.

Reading:

Tristan Tomilin, Meng Fang, Yudi Zhang, and Mykola Pechenizkiy. COOM: A Game Benchmark for Continual Reinforcement Learning. NeurIPS 2023

Tristan Tomilin, Meng Fang, and Mykola Pechenizkiy. HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agents. ICLR 2025.

Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research 16.1 (2015): 1437-1480.

Khetarpal, Khimya, et al. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research 75 (2022): 1401-1476.

Details
Student
SG
Sarthak Goswami
Supervisor
Tristan Tomilin
Secondary supervisor
Thiago Simão