Project: Causal Discovery for Offline Model-Based Reinforcement Learning
Description
Reinforcement Learning (RL) has proven effective in a variety of complex decision-making tasks. However, traditional RL requires extensive online interactions, making it costly and, in some domains, impractical due to constraints on safety, time, or resource availability. Offline RL, which relies solely on pre-collected datasets, presents a promising alternative by enabling learning without direct environment interaction. Model-based approaches further enhance this by using a learned dynamics model to simulate the environment, enabling efficient exploration and decision-making.
However, learning accurate dynamics models from offline data is challenging due to issues like distributional shift and spurious correlations in the datasets. Advances in causal modeling offer a promising solution by capturing true cause-effect relationships, making models more robust and generalizable. Incorporating causality allows us to build dynamics models that are not only predictive but also resilient to out-of-distribution scenarios, leading to better decision-making in offline RL settings.
In this project, we aim to advance offline RL research by introducing causally-informed dynamics models that enhance robustness and generalization. This involves conducting a comprehensive literature Review on Model-Based Offline RL and Causal Modeling, development of causal dynamics model tailored for offline RL settings, and model training and evaluation framework using standard offline RL benchmarks.
References:
- A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
- MOPO: Model-based Offline Policy Optimization
- RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning
- Offline Reinforcement Learning with Causal Structured World Models
- Learning Dynamic Attribute-factored World Models for Efficient Multi-object Reinforcement Learning
Details
- Supervisor
-
Maryam Tavakol
- Secondary supervisor
-
Devendra Dhami
- Interested?
- Get in contact