Demonstration and offset augmented meta reinforcement learning with sparse rewards

Demonstration and offset augmented meta reinforcement learning with sparse rewards

Abstract This paper introduces DOAMRL, a novel meta-reinforcement learning (meta-RL) method that extends the Model-Agnostic Meta-Learning (MAML) framework. The method addresses a key limitation of existing meta-RL approaches, which struggle to effectively use suboptimal demonstrations to guide train...

Full description

Saved in:
Bibliographic Details
Main Authors: Haorui Li, Jiaqi Liang, Xiaoxuan Wang, Chengzhi Jiang, Linjing Li, Daniel Zeng
Format: Article
Language:English
Published: Springer 2025-02-01
Series:Complex & Intelligent Systems
Subjects:
Meta learning
Reinforcement learning
Sparse reward
Suboptimal demonstration
One-shot learning
Imitation learning
Online Access:https://doi.org/10.1007/s40747-025-01785-0
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Abstract This paper introduces DOAMRL, a novel meta-reinforcement learning (meta-RL) method that extends the Model-Agnostic Meta-Learning (MAML) framework. The method addresses a key limitation of existing meta-RL approaches, which struggle to effectively use suboptimal demonstrations to guide training in sparse reward environments. DOAMRL effectively combines reinforcement learning (RL) and imitation learning (IL) within the inner loop of the MAML framework, with dynamically adjusted weights applied to the IL component. This enables the method to leverage the exploration strengths of RL and the efficiency benefits of IL at different stages of training. Additionally, DOAMRL introduces a meta-learned parameter offset, which enhances targeted exploration in sparse reward settings, helping to guide the meta-policy toward regions with non-zero rewards. To further mitigate the impact of suboptimal demonstration data on meta-training, we propose a novel demonstration data enhancement module that iteratively improves the quality of the demonstrations. We provide a comprehensive analysis of the proposed method, justifying its design choices. A comprehensive comparison with existing methods in various stages (including training and adaptation), using both optimal and suboptimal demonstrations, along with results from ablation and sensitivity analysis, demonstrates that DOAMRL outperforms existing approaches in performance, applicability, and robustness.
ISSN:2199-4536
2198-6053

Similar Items