Imitation learning and behaviour cloning

Behaviour cloning as supervised learning#

Given a dataset of state-action pairs {(s_i, a_i)} collected from an expert policy pi*, behaviour cloning solves:

pi_hat = argmin_pi   (1/N) * SUM_i  L( pi(s_i), a_i )

for a chosen loss L — squared error for continuous actions, cross-entropy for discrete ones. This is standard empirical risk minimisation, with no notion of the environment's dynamics anywhere in the objective.

Why error compounds: the horizon bound#

Ross and Bagnell (2010) showed the central negative result. Let epsilon be the per-step probability the cloned policy takes an action the expert would not have taken, and T the task horizon. Under behaviour cloning's training distribution (the expert's own state visitation), the expected number of mistakes over a rollout is bounded by:

E[ total cost ]  <=  O( epsilon * T^2 )

The T^2 term is the crux. A mistake at step t pushes the agent into a state outside the training distribution, for the remaining T - t steps. The per-step error can then compound further — unlike the naive O(epsilon * T) you would get if every step's state distribution matched training.

DAgger#

Ross, Gordon, Bagnell (2011), A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, introduces DAgger (Dataset Aggregation) to close this gap:

1. Train pi_1 on initial expert demonstrations D_1
2. For i = 1 .. N:
     Roll out pi_i in the environment, collecting visited states S_i
     Query the expert for the correct action at each state in S_i
     D_{i+1} = D_i UNION (S_i, expert actions)
     Train pi_{i+1} on D_{i+1}

This achieves an O(epsilon * T) regret bound instead of O(epsilon * T^2), because training data now includes states the policy itself visits, not only states the original expert visited. The practical cost is real: it needs an expert (often a human) available on-line, throughout training, not only once upfront.

Inverse reinforcement learning#

A related but distinct family infers a reward function consistent with the expert's demonstrations, then solves the ordinary RL problem with the inferred reward. Ng and Russell (2000) formalised inverse RL. Ho and Ermon (2016), Generative Adversarial Imitation Learning, reformulate it as a GAN-style adversarial game between a policy and a discriminator distinguishing expert from policy trajectories — avoiding an explicit reward reconstruction step.

Current state#

Large-scale robot imitation learning has shifted toward transformer-based policies. These train on mixed teleoperation datasets across many tasks and robot embodiments — for instance the Open X-Embodiment dataset (Padalkar et al., 2023) — rather than one policy per task trained from scratch. Distribution shift remains an open problem at this scale; broad, diverse demonstration coverage is the primary practical mitigation, alongside DAgger-style iterative correction where an expert remains available.

Key references#

• Pomerleau, D. (1989). ALVINN: An Autonomous Land Vehicle in a Neural Network. NeurIPS — an early behaviour-cloned driving system.
• Ross, S., Bagnell, D. (2010). Efficient Reductions for Imitation Learning. AISTATS.
• Ross, S., Gordon, G., Bagnell, D. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS.
• Ho, J., Ermon, S. (2016). Generative Adversarial Imitation Learning. NeurIPS.
• Padalkar, A. et al. (2023). Open X-Embodiment: Robotic Learning Datasets and RT-X Models.

What to learn next#

• What is reinforcement learning? — the framework DAgger and inverse RL both build on.
• Vision-language-action models — where large-scale imitation learning is currently headed.
• Grasping and manipulation — a task family this technique is heavily applied to.