IEEE International Conference on Robotics and Automation 2024
In-Context Imitation Learning via Next-Token Prediction
Fig. 1: In-Context Robot Transformer (ICRT): A robot foundation model with in-context imitation learning capabilities. ICRT performs next-token prediction on large-scale sensorimotor trajectories. At inference time, it takes raw sensorimotor trajectories of human teleoperation demonstrations as prompts, enabling the model to execute new tasks with real-time continuous control, without requiring fine-tuning.
Abstract
We explore how to enable in-context learning capabilities of next-token prediction models for robotics, allowing the model to perform novel tasks by prompting it with human teleop demonstration examples without fine-tuning. We propose In-Context Robot Transformer (ICRT), a causal transformer that performs autoregressive prediction on sensorimotor trajectories, which include images, proprioceptive states, and actions. This approach allows flexible and training-free execution of new tasks at test time, achieved by prompting the model with demonstration trajectories of the new task. Experiments with a Franka Emika robot demonstrate that the ICRT can adapt to new tasks specified by prompts, even in environment configurations that differ from both the prompts and the training data. In a multi-task environment setup, ICRT significantly outperforms current state-of-the-art robot foundation models on generalization to unseen tasks. Code, checkpoints and data are available on https://icrt.dev.
Download publicationAssociated Researchers
Letian Fu
UC Berkeley
Huang Huang
UC Berkeley
Gaurav Datta
UC Berkeley
Lawrence Yunliang Chen
UC Berkeley
Will Panitch
UC Berkeley
Fangchen Liu
UC Berkeley
Ken Goldberg
UC Berkeley
Related Publications
2023
Safe Self-Supervised Learning in Real of Visuo-Tactile Feedback Policies for Industrial InsertionThis research introduces a robotics method that reduces human input…
2024
ASAP: Automated Sequence Planning for Complex Robotic Assembly with Physical FeasibilityA physics-based planning approach for automatically generating…
2024
Bridging the Sim-to-Real Gap with Dynamic Compliance Tuning for Industrial InsertionA novel framework for robustly learning manipulation skills…
2023
Constrained-Context Conditional Diffusion Models for Imitation LearningA diffusion model policy for solving 6-DoF robotic manipulation tasks…
Get in touch
Something pique your interest? Get in touch if you’d like to learn more about Autodesk Research, our projects, people, and potential collaboration opportunities.
Contact us