Learning Dense Reward with Temporal Variant Self-Supervision

Reinforcement learning (RL) is gaining momentum in solving complex real-world robotics problems. One challenging category is contact-rich manipulation tasks. The success of RL in these scenarios depends on a reliable reward system. While this genre of problems is marked by rich, high dimensional, continuous observations, it is typically hard to come up with a dense reward that can harness such richness to guide RL training. The conventional way of using sparse, boolean rewards (e.g., 1 if the task is successfully completed and 0 otherwise) is often challenging and inefficient. The difficulty has led to the practice of reward engineering, where rewards are hand-crafted based on domain knowledge and trial-and-error. However, such solutions often require extensive robotics expertise and can be task-specific.

In this research, we propose an end-to-end learning framework that can extract dense rewards from multimodal observations. The reward is learned in a self-supervised manner by combining one or two human demonstrations with a physics simulator, and can then be directly used in training RL algorithms. We evaluate our framework in two contact-rich manipulation tasks, joint assembly and door-opening tasks.

There are two main contributions in this paper: 1) a temporal variant forward sampling (TVFS) method that is more robust and cost-efficient in generating samples from human demonstrations for contact-rich manipulation tasks, 2) a self-supervised latent representation learning architecture that can utilize sample pairs from TVFS.