Behaviour cloning

The chapter overview said that most learned policies on a robot arm today are built from models that copy a person. This page explains behaviour cloning, which is the simplest of those models, and the base for the two pages that follow. A person does the task many times while the robot records everything, and a neural network then learns to copy what the person did at each moment.

The page answers these questions, in the order you would meet them while building such a policy. What goes into a behaviour cloning model, and what comes out? How does the model learn, and how much recording does it need? How do you tell one policy to do more than one task? And which real models do this, and which of them should you start with?

It is for a reader who has read the chapter overview, and who knows what a policy, an observation and an action are. If you have not read how a model learns, then read that page first, because this one uses its idea of a model making a guess, being told the right answer, and adjusting.

Contents#

1. What behaviour cloning is
2. What goes in and what comes out
3. How it works inside
4. How it is trained
5. Telling the policy what to do: goals and many tasks
6. Well-known models of this kind · 6.1 ALVINN, the method with nothing added · 6.2 robomimic, if you want plain behaviour cloning · 6.3 ACT in LeRobot, the one to start with · 6.4 DAgger, which is now a command · 6.5 SmolVLA, if fifty recordings are not enough · 6.6 How to choose
7. Where this is going
8. Where to read next

1. What behaviour cloning is#

The introduction called behaviour cloning the simplest way to teach an arm, so this section says what the two words themselves mean. Behaviour cloning means training a network to copy what a person did in each situation.

Here is an everyday example of the same thing. Imagine you are learning to make tea by watching a friend, and you never ask them to explain the rules. Instead you watch what they do at each moment: the kettle has boiled, so they pour, and the cup is full, so they stop. Later you make tea yourself by doing what they did in the same situation.

Behaviour cloning works the same way, but the situation is what the robot's cameras see and where its joints are. The thing to copy is the move that the person made next from that situation.

A few words are needed for this, and all three of them come back later on the page.

• A demonstration is one recording of a person doing the task once, from start to finish, and people also call it an episode.
• Teleoperation means a person controls the robot from a distance, with some kind of controller, and it is the most common way to record demonstrations. The robot's own sensors then record exactly what the robot did.
• The demonstrator is the person who does the task during recording.

The word "cloning" means making a copy, so the name says that the model tries to make a copy of the demonstrator's behaviour.

2. What goes in and what comes out#

Section 1 said what behaviour cloning copies, and this section says which numbers the model actually reads and writes. A behaviour cloning model is a policy, so it turns an observation into an action.

For a robot arm, the observation usually has two parts. The first part is one or more camera pictures, taken at this moment, and the second part is the arm's current joint angles, read from the sensors in each joint.

The action is what the arm should do next, and it is usually written in one of two ways.

• The joint angles the arm should move to next, which is six numbers for a six-joint arm.
• How far the gripper should move, forwards and back, left and right, up and down, and how much it should turn.

Either way, one more number says how open or closed the gripper should be.

A demonstration is recorded as a long list of moments, often 10 to 50 moments a second. At each moment the recording holds the pictures, the joint angles, and the move the person made next. The picture below shows four such moments, from a recording of someone picking up a mug.

Each column of the drawing is one moment. The picture and the joint angles are the question, and the green box is the answer that the network must learn to give.

This is the key point of the whole page. The recording is cut into many small question-and-answer pairs, and each pair says "in this situation, the person did this". So the network never sees the task as a whole, and it only ever sees many separate moments.

3. How it works inside#

Section 2 described the numbers going in and out, and this section follows them through the network itself. A behaviour cloning model is an ordinary neural network with three parts, and the data flows through them in order.

1. The picture part. The camera picture goes into a network that is good at pictures, which is often a convolutional neural network (CNN), a kind of network that looks at small patches of the picture at a time. A common choice is ResNet, which is a well-known CNN. ResNet is a part inside the policies named in section 6 rather than a policy of its own, which is why it is not one of the rows there. This part turns the picture into a list of a few hundred numbers, and those numbers describe what is in the picture, such as where the mug is. Inside a neural network explains how this works.
2. Joining. The list of numbers from the picture is joined onto the joint angles, so that one single list now describes the whole situation.
3. The deciding part. A few more layers of the network turn that list into the action numbers, which are the next joint angles and the gripper opening.

Training works as it does for any other network. For each question-and-answer pair the network makes a guess, and that guess is compared with the move the person actually made. The difference between them is called the loss. For example, if the network guessed 62 degrees and the person moved to 63 degrees, then the loss for that joint is 1 degree. The training program then nudges every number inside the network a tiny amount, so that the next guess is a little closer. It does this over and over, for every pair, many times.

When training is finished, the network is run on the robot, where it takes the live camera picture and joint angles and gives an action. The arm makes that move, and then the network takes the next picture and gives the next action.

Some versions also give the network the last few pictures, and not just the current one. This helps it tell whether the arm is moving up or down, which a single picture cannot show. The recurrent policy called BC-RNN, in section 6.2, is this idea written out: it remembers the last few moments instead of judging each one alone.

These three parts are the shape that every model in section 6 is built on, and each one there changes a part or adds to it. ALVINN, in section 6.1, is the shape at its simplest, because its three layers take the picture and give back the steering direction with nothing added. The plain behaviour cloning in robomimic, in section 6.2, is the same shape with the surrounding program written for you. ACT, in section 6.3, replaces the deciding part with a transformer that gives back a run of future commands instead of one, and the next page is about that change. SmolVLA, in section 6.5, adds a further input, which is a sentence saying which task to do, and it arrives with its numbers already trained. DAgger, in section 6.4, changes none of this, because it alters where the training pairs come from rather than the network.

4. How it is trained#

Section 3 described the network, and this section describes the data it is trained on. The data is a set of demonstrations of one task, and there are three common ways to record them.

• Leader and follower arms. The person moves a small copy of the robot arm by hand, and the real arm copies each joint angle. This is how the ALOHA system records its data, and the next page describes it.
• A handheld controller or a virtual reality headset. The person moves a controller in the air, and the robot's gripper follows it.
• Moving the arm by hand. Some arms let you push them around directly while they record, and this is called kinesthetic teaching. It is simple, but your hand is then in the camera picture, and the robot will not see a hand when it runs on its own.

How many demonstrations are needed for one task? For one narrow task, such as picking up one mug from different places on a table, people usually record from a few dozen to a few hundred. The more the task varies, the more demonstrations it needs, and the page where the data comes from explains how people collect data at a larger scale.

Training a small behaviour cloning policy on one graphics card usually takes hours rather than days. So the recording usually takes longer than the training does.

It helps a great deal if the demonstrations are consistent with each other. If the demonstrator sometimes pauses, sometimes hesitates, and sometimes goes a different way, then the network gets mixed answers for the same question. A study called robomimic, which is listed below, found that data from one skilled demonstrator trained better policies than data from a mix of more and less skilled demonstrators.

5. Telling the policy what to do: goals and many tasks#

So far the policy has been given only what it sees, and it then does the one task it was trained on. This section shows how to tell a policy which task to do, so that one policy can do several of them. It also shows a method that turns aimless recordings into useful training data.

A goal-conditioned policy is a policy that is given the goal as part of its observation, and "conditioned" here means "depends on". So the same picture of the table can lead to different moves, depending on which goal is given.

Here is an everyday example of the difference. A taxi driver at a crossroads does not always turn left, because where they turn depends on where the passenger asked to go. The road is the same, but the destination changes the answer. So a goal-conditioned policy matches the driver who is told the destination, while a plain policy matches the driver who has only ever driven one route.

Three ways to give the goal#

Now that the goal is part of the observation, the next question is how the goal is written down. There are three common ways to tell a policy what the goal is, and the picture below shows each one, for a scene with a mug and two bins.

Each of the three goals says the same thing, which is that the mug should end up in the blue bin. The policy gets one of them, next to the live camera picture.

• A goal picture. This is a photo of how the scene should look at the end, and it goes through the same kind of seeing network as the camera picture. It needs no words, and it can show exactly where something should go. But someone has to make the photo, which usually means doing the task once first.
• A task number. Each task gets a number, and the policy is given that number. The number is usually written as a one-hot list, which is a list with one place per task, all zeros except a 1 in the place of the chosen task. So [0, 1, 0, 0] means "task 2 of 4". This is the simplest of the three ways, but the policy cannot do a task that has no number yet.
• A sentence, such as "Put the mug in the blue bin". A pretrained language model turns the sentence into a list of numbers first, and RT-1 used a model called the Universal Sentence Encoder for this. A sentence can put known words together in a new way, such as "the red bin" when training only said "red cup" and "blue bin". But whether the policy then does the right thing is not guaranteed. The language models chapter takes this much further.

Why the goal must be an input#

The three ways above all feed the goal into the policy, so it is worth asking why the goal has to go in at all. Why not train one policy per task, or train one policy on all the tasks without telling it which one?

The second idea fails for the reason given in two good ways become one bad way. If the same picture sometimes leads to the red bin and sometimes to the blue bin, then a plain policy learns the average of the two. The picture below shows this happening with a real fit.

The faint lines are 40 recorded demonstrations, of which half go to the red bin at (−25, 45) cm and half go to the blue bin at (25, 45) cm. The script for this page fitted the simplest possible policy to them, with the method of least-squares fitting. The policy says how far to move, given where the mug is. On the left, the policy was given only where the mug is, so it ends at (−1.1, 44.8) cm, between the bins and in neither of them. On the right it was also given a one-hot goal, and when it is asked for red it ends at (−25.1, 45.0) cm, while asked for blue it ends at (25.0, 44.7) cm.

So the goal removes the mixed answers, because each situation then has only one right move again. A real policy is a large network rather than a straight-line fit, but the effect is the same.

One policy for many tasks#

Once the goal is an input, one policy can be trained on the recordings of many tasks at once. This is called multi-task training.

It helps in two ways. The first is sharing, because picking up a mug, picking up a cup and picking up a bowl all start with the same reach and the same grip. The policy learns those shared parts from all the recordings together, so each task needs fewer recordings of its own. The second is convenience, because there is one policy to train, store and run, instead of dozens of them.

It also has a cost. Tasks that look alike can get mixed up, so a policy trained on many tasks is sometimes a little worse at each one than a policy trained on that task alone. The usual answer to that is more data, and a larger network.

BC-Z, from 2021, trained one policy on about a hundred tasks, and gave it the goal as a sentence or as a video of a person doing the task. RT-1, from 2022, went further, to 700 tasks.

Hindsight relabelling#

Multi-task training needs recordings of every task, and recording them all is slow. So there is a method that turns recordings which failed, or which had no task at all, into good training data. It is called hindsight relabelling, because "hindsight" means looking back after something has happened.

The idea is simple. A recording that was meant to reach one place, but reached another, is still a perfect demonstration of how to reach the place it did reach. So you change its label: you take the place where the recording ended, and you call that its goal.

On the left are eight attempts to reach the star. The script made them with some randomness, so only one of the eight ends within 5 cm of the star, which means that seven of them are useless as demonstrations of "reach the star". On the right, each path's own end point is its goal. So all eight are now correct demonstrations, each one of reaching its own goal.

This is how people use unscripted play data. A person moves the robot around freely for a while, opening drawers, pushing blocks and picking things up, with no task in mind. Afterwards the recording is cut into short pieces, and the last picture of each piece becomes that piece's goal picture. So one hour of play gives thousands of goal-conditioned examples, without anyone deciding the tasks in advance. This was shown by the paper Learning from Play (Lynch and others, 2019), and the same idea in reinforcement learning is called Hindsight Experience Replay, from 2017.

Where it fails, and the tools#

Goals and multi-task training bring faults of their own, and these are the common ones, each with the sign you would see.

• The policy ignores the goal. It does the same thing whatever you ask for, and this happens when each scene in the training data was almost always recorded with one task. The policy can then guess the task from the picture alone, so it never learns to read the goal. The fix is to record different tasks in the same scene.
• A goal picture says too much. The photo also shows where every other object is, and the light at the time, so the policy may try to match all of it. People therefore use goal pictures that show only the part that matters, or they use a sentence instead.
• A task number cannot grow. A new task needs a new number, and the policy has never seen it. So only a sentence or a picture can describe a task the policy was not trained on.
• Relabelled data is only as good as the play. The paths in play data wander, so a policy trained on them reaches the goal, but not always by the shortest way. People therefore mix in some clean demonstrations.

Several tools support all of this directly. LeRobot stores a task sentence with each recording, and its larger policies, such as SmolVLA and π0, take that sentence as input. The vision-language-action models page describes those policies. Book 3's foundation models document describes π0.7, which is given a sentence, a goal picture and labels describing how to do the task, all at once.

6. Well-known models of this kind#

Section 4 described how the data is recorded. This section names the behaviour cloning models worth knowing about, says which one to reach for first, and gives for each one the shortest command or program that does something real with it.

Read the table as a filter rather than as a ranking. The left column names the model and says how much use it gets. The right column holds everything you need to filter on: what it is best at, its size, its licence, whether it trains on an Apple Silicon Mac with no separate graphics card, and when to pick it. Find the row that matches the hardware and the recordings you have, then read that model's sub-section. A size is the number of trainable values the project itself states, and it says not stated where no project document gives a figure. The answers about the Mac come from LeRobot's own compute hardware guide, which groups policies by the video memory they need to train at a batch size of eight.

6.1 ALVINN, the method with nothing added#

Historical. Size not stated, a laptop, and no licence at all, because no code was released with the paper.

ALVINN stands for Autonomous Land Vehicle In a Neural Network, and Dean Pomerleau published it at Carnegie Mellon University in 1988. Its own abstract describes a three-layer network that takes pictures from a camera and a laser range finder and produces the direction the vehicle should steer. One observation goes in, one action comes out, and the training signal is the recorded answer.

The one idea in ALVINN is that the network is the whole program. The paper's introduction says why that was worth trying. Road-following systems of the time were built as a chain of hand-written image processing steps, and they worked under some conditions and failed under others, because the processing they did never changed from one situation to the next. ALVINN replaced the whole chain with a single network trained on recorded pairs. Nothing inside it names a road edge or a lane, and that decision is the one every model further down this page inherited.

Inside, the network is as plain as the idea. Its input is a grid of 30 by 32 numbers taken from the blue part of the camera picture, a grid of 8 by 32 numbers from the laser range finder, and one further number saying whether the road was lighter or darker than its surroundings in the previous picture. Each of those numbers has its own connection to each of the 29 units in the single middle layer. There is no convolution and no sharing of weights, which is the difference from the picture part described in section 3. A convolutional network looks for the same small pattern everywhere in the picture, while ALVINN learns separately what belongs at each position of its grid.

The output side is the part worth carrying away, because it is not a number. ALVINN has 45 output units in a row. The middle one means drive straight ahead, and the units to either side mean progressively sharper left and right turns. Training does not ask one unit to fire. It asks for a hill of activation centred on the correct turn and falling away on both sides. When the vehicle drives, the steering is read from whichever unit is most active. So ALVINN chooses an answer instead of computing one, and that distinction is what the rest of this page turns on. The paper does not make the following argument, but it follows from the shape of the output. A network that gives back a number is pulled towards the average of the recordings it was shown, so two recordings that disagree produce a third answer between them that nobody demonstrated. A network that reads off the highest of 45 positions grows two hills instead and takes the taller one. The spread of bell curves in section 6.2 and the diffusion policy two pages on arrive at the same place by other routes.

What that design bought was a system that could be trained rather than programmed, and what it cost is written into the paper's own training section. ALVINN was trained on pictures from a road generator rather than on real roads, because collecting enough real pictures was difficult, and because changing something such as the camera's orientation would have meant collecting the whole set again. That sentence from 1988 is the oldest statement of the cost every reader of this book meets, which the chapter overview puts as a policy learning the view and not the world. The one feedback number is also all the memory the network has, so it cannot tell whether it is moving towards the road or away from it, and that is the limit section 6.2 fixes properly.

You would not pick ALVINN, and there is nothing to pick, because no code was released with it. What you would use instead is robomimic in section 6.2, which is the same method with the parts a modern project needs already written. The reason to read about ALVINN is that every policy further down this page is this network with pieces added, so once section 6.3 puts a transformer in the middle of it, you can still see the picture going in and the command coming out. What the shape cost the field is the fault the next page exists to repair: a policy that chooses one move at a time drifts away from its recordings within a second or two.

There is no library for ALVINN, so the shortest honest code is ALVINN written again in PyTorch over a modern recording. The data comes from LeRobot, whose LeRobotDataset class reads a recording and behaves like an ordinary PyTorch dataset.

import torch
from torch import nn
from lerobot.datasets import LeRobotDataset

# A real recording on the Hugging Face Hub: one SO-101 arm picking objects up.
dataset = LeRobotDataset("lerobot/svla_so101_pickplace")
loader = torch.utils.data.DataLoader(dataset, batch_size=64, shuffle=True)

# The recording states the shape of every field, so the network is sized from it.
state_dim = dataset.features["observation.state"]["shape"][0]
action_dim = dataset.features["action"]["shape"][0]
net = nn.Sequential(nn.Linear(state_dim, 256), nn.ReLU(), nn.Linear(256, action_dim))
optimizer = torch.optim.Adam(net.parameters(), lr=1e-4)

for batch in loader:
    # One behaviour cloning step: guess, compare with the person's action, adjust.
    loss = nn.functional.mse_loss(net(batch["observation.state"]), batch["action"])
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Those four lines in the loop are the whole method, which shows that the difficulty of such a project is not in the training step. The library gave you the file format, the loading and the field shapes. What this network lacks is what the sub-sections below supply: it reads only observation.state, the arm's own joint positions, and never looks at the pictures, so it learns the average path of the recordings instead of reacting to where the object is. To match even ALVINN you would add an encoder that turns a picture into numbers, and the rescaling of every input and output into the range a network expects.

6.2 robomimic, if you want plain behaviour cloning#

Most used in 2026, for the one job of running plain behaviour cloning and comparing against a published number. Size not stated, a laptop, MIT.

robomimic came out of the Stanford Vision and Learning Lab in 2021 with a study of six offline learning methods on five simulated and three real manipulation tasks, using recordings of deliberately different quality. The library is still maintained, and its behaviour cloning file holds eight variants, including the plain one, a recurrent one called BC-RNN that lets the policy remember the last few moments, and a transformer one.

The one idea in robomimic is that the design choices are the subject. The study asked which of them decide whether a policy works, so the library does not ship one network. It ships one training program whose behaviour cloning file builds those eight policies by answering two questions in different ways, and you answer them with settings in a configuration file rather than by writing code. The first question is whether the policy sees only the present moment or also the recent past, and the recent past can be held either by a recurrent network or by a transformer. The second question is whether its answer is one action or a description of several possible actions. Nothing else about the run changes when you change one answer, which is what makes a comparison between two settings mean something.

Turn the first switch, algo.rnn.enabled, and the deciding part described in section 3 is replaced by a recurrent network, which is a network that keeps a few numbers from one moment and reads them back at the next one. Those numbers are its memory, so its answer now depends on what it saw a moment ago and not only on what it sees now. That is the fix for the limit section 3 named, where a single picture cannot show whether the arm is moving up or down. The variant is called BC-RNN, and robomimic's default unrolls that memory over ten steps while training.

The second switch, algo.gmm.enabled, is the one that matters most on this page. With it off, the network gives back a set of action numbers, and training shrinks the distance between those numbers and the recorded ones. With it on, the network gives back the shape of a spread of actions instead: the centre and the width of each of a few bell curves, five of them by default, together with how likely each one is. Training then raises the probability that this spread gives to the action the person actually took, rather than shrinking a distance to it. The consequence is the single most useful idea about models that copy a person. If half the demonstrations go left round an obstacle and half go right, a network that must give back one set of numbers is pushed towards the point between them, which is the obstacle, while a spread of five curves can put one curve on the left route and another on the right and keep both. At run time robomimic draws an action from that spread, and its low_noise_eval setting narrows the curves first, so the draw lands near one centre instead of between two centres.

What the switches buy you is the chance to test one of these ideas instead of believing it, and the cost is that you test it on somebody else's robot. Nothing here predicts a chunk, so every one of the eight variants still chooses one move at a time and still drifts, which is what the next page is about. On a robot arm the difference shows up when your own policy wavers or stalls at the same point of the task every attempt. Before you record another hundred demonstrations, robomimic lets you check on its recordings, in an afternoon, whether memory or a spread of answers is the piece you were missing, and then look for the same option in the policy you actually run.

You would pick robomimic over LeRobot for this one job because LeRobot ships no plain behaviour cloning policy at all. Its list of policies, read from its source on 3 October 2026, is act, diffusion, vqbet, tdmpc, smolvla, pi0, pi0_fast, pi05, groot, eo1, evo1, xvla, wall_x, molmoact2, multi_task_dit, flux3, lawam, lingbot_va, vla_jepa, fastwam and gaussian_actor. Every one of those is behaviour cloning with something added.

What it costs you is the shape of the project rather than the computing. It expects its recordings as HDF5 files in its own layout, which is not the LeRobot format, so your own recordings need converting, and it is driven by a configuration file rather than by flags. The thing that most often goes wrong is the installation, because it expects a particular pairing of robosuite and MuJoCo, which is why Book 3's glossary says to install it from GitHub rather than from the Python package index.

The commands below are from robomimic's own getting started page, and they need no robot.

# Install from the repository rather than from the package index.
git clone https://github.com/ARISE-Initiative/robomimic
pip install -e robomimic

# One of the study's own datasets: lifting a block, recorded by one skilled person.
python robomimic/robomimic/scripts/download_datasets.py --tasks lift --dataset_types ph

# Train plain behaviour cloning. The template is the whole configuration.
python robomimic/robomimic/scripts/train.py \
  --config robomimic/robomimic/exps/templates/bc.json \
  --dataset datasets/lift/ph/low_dim_v141.hdf5

The library gives you the network, the picture encoder, the rescaling, the training loop and a published success rate to compare against. You supply the configuration file, and the setting in it to change first is algo.rnn.enabled, from false to true, which switches the same file to BC-RNN, because whether the network sees past moments is one of the details the study found to matter. What you cannot get from robomimic is a policy for your own arm, because its recordings are of a simulated arm in a simulated room.

6.3 ACT in LeRobot, the one to start with#

Most used in 2026. Size s, a small card, Apache-2.0 for LeRobot's code and MIT for the original, and no weights to download, because the policy you run is one you trained yourself.

ACT stands for Action Chunking with Transformers, published in 2023 with the ALOHA paper. It is behaviour cloning that predicts a short run of future commands at each decision instead of one command, and the next page is about how it does that. LeRobot's own page for it calls it "the first model we recommend when you're starting out", and says it often reaches a high success rate with 50 demonstrations.

The one idea in ACT is that the answer is a stretch of time rather than a moment. All eight robomimic variants in section 6.2 answer the question "what should the arm do now", and BC-RNN answers that question better by remembering the last few moments. ACT changes the question to "what should the arm do for the next two seconds", and gives back about a hundred commands in one pass.

Inside, the picture part and the joining stay as section 3 described them, and the deciding part is what changes. A transformer takes the place of the last few layers, and its output side holds one empty slot for each future moment, where each slot knows only which moment it stands for. Attention fills those slots in. Every slot may look at every patch of every camera picture, at the joint positions, and at the other slots, so the hundredth command is produced in the same pass as the first. One small layer then turns each filled slot into a full set of joint targets. The number of decisions taken during the task therefore falls by roughly the length of the chunk, and that is the whole of the repair, because the drift this page keeps mentioning comes from the policy's own small errors carrying it into situations nobody recorded, and most of the steps where that could happen have gone.

ACT also does something about demonstrations that disagree, and it is worth comparing with the spread of bell curves in section 6.2. While ACT trains, a second transformer reads the chunk the person actually made, together with the joint positions, and squeezes how it was done this time into 32 numbers. The main network is handed those 32 numbers as an extra input during training, so it is not forced to explain the difference between two demonstrations from the picture alone. When the trained policy runs there is no person and no recorded chunk, so the 32 numbers are all set to zero, which stands for the typical way of doing the task. So robomimic keeps several routes and draws one of them at run time, while ACT keeps the variation out of the way during training and then asks for the middle of it afterwards. LeRobot lets you see what that is worth on your own data, because --policy.use_vae=false sets those numbers to zero during training as well, which turns ACT into plain prediction of one chunk.

What the chunk buys is smooth motion on a careful task from about fifty demonstrations, and what it costs is reaction time, because an arm carrying out a chunk is not looking at the pictures. The zeroing at run time costs you the rest. A single typical chunk is one answer, so recordings that disagree about the route are still averaged into one route that may go straight through the obstacle. On a robot arm the difference from plain behaviour cloning shows up on a slow careful approach, such as easing a plug into a socket, where a policy choosing one command at a time hesitates and corrects itself into the side of the socket, while ACT commits to the whole approach. The difference in the other direction shows up if two people recorded your demonstrations and went about the task differently, and that is the case the diffusion policy two pages on was built for.

You would pick ACT over plain behaviour cloning from robomimic for the reason just given: a policy that chooses one command at a time drifts, and predicting a run of commands at once cuts the number of decisions and therefore the drift. Book 3's page on learned methods quotes the ACT paper's own comparison, which goes from 1 per cent success when predicting one command at a time to 44 per cent when predicting a hundred, averaged over simulated tasks. The paper's real-robot claim is 80 to 90 per cent on six fine manipulation tasks from ten minutes of demonstrations.

What it costs you is a recording session and nothing else, because ACT is in the base LeRobot installation and needs no extra dependency. The thing that most often goes wrong is expecting somebody else's trained ACT policy to work for you. There is no transferable one, because the output layer has one number per joint of the arm it was trained on, so a policy trained on a seven-joint arm cannot even be loaded for a six-joint one.

Training is a command rather than a program, because LeRobot reads the number of joints and the number of cameras from your recording and sizes the network to fit.

# Train. The network is sized from the dataset, so nothing here mentions joints.
lerobot-train \
  --policy.type=act \
  --dataset.repo_id=${HF_USER}/my_dataset \
  --output_dir=outputs/train/act_my_dataset \
  --job_name=act_my_dataset \
  --policy.device=cuda          # use mps on an Apple Silicon Mac

# Run the trained policy on the real arm for sixty seconds, recording nothing.
lerobot-rollout \
  --strategy.type=base \
  --policy.path=outputs/train/act_my_dataset/checkpoints/last/pretrained_model \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
  --duration=60

LeRobot gives you the network from the paper, the training loop, the rescaling, the checkpoints and the control loop that runs the policy on the arm at a fixed rate. You supply the recording, and that is the whole of the work. The decision inside it that costs the most later is what you vary while recording, because the range of object positions, the lighting and the camera placement in your recordings become the limits of the finished policy.

6.4 DAgger, which is now a command#

Worth betting on, because the step it describes stopped being a research project and became one flag on a command. No size and no machine of its own, because DAgger is a way of collecting data rather than a network, and Apache-2.0 as part of LeRobot.

DAgger stands for Dataset Aggregation, and Ross, Gordon and Bagnell published it in 2011. You let the trained policy drive, you take over when it goes wrong, and you add your corrections to the training set.

The one idea in DAgger is that the fault in behaviour cloning is in the data and not in the network. Every model above this one is trained on pairs that were recorded while a person was driving, so every situation in the training set is a situation a person got into. A trained policy drives itself into slightly different situations, because its own small errors take it a little off the recorded path, and nobody ever showed it what to do there. Changing the network does not change that, which is why DAgger sits alongside ACT and SmolVLA rather than competing with them.

What changes is a loop around the training rather than anything inside it. You train a policy on the recordings you have, you run it, and the pairs you add are the situations it reached together with the action you would have taken in them. Those new pairs are added to the set you already had, which is where the word aggregation in the name comes from, and the next policy is trained on the whole grown set. The paper's own claim is about this loop and not about any network: repeated this way, it finds a policy that performs well in the situations that the policy itself produces, which is exactly the thing plain behaviour cloning promises nothing about.

What the loop buys is coverage in the one place where it was missing, and its own limit is that it repairs only one of the two faults this page keeps returning to. A correction tells the policy what to do somewhere it had never been, so it works against drift. It does nothing against disagreement, because if your recordings already disagreed about the route then your corrections will disagree in the same way, and the averaging stays. On a robot arm the difference shows up with a policy that reaches for the drawer handle and stops two centimetres short every attempt. More demonstrations of the same kind give the policy more of what it already had, because a person never stops two centimetres short. Twenty corrections taken from where it does stop give it the one thing it lacks.

Book 3's page on interactive imitation states that a few dozen corrections gathered this way are often worth several hundred fresh demonstrations. The reason to bet on it now rather than in 2011 is that LeRobot ships it: its deployment page lists dagger as one of the strategies of the lerobot-rollout command, it alternates between the policy driving and you driving, and it marks every frame you drove with intervention=True.

What it costs you is your attendance. You have to be at the robot with the leader arm in your hands, so the strategy requires a teleoperator to be configured, and the collecting cannot be left running overnight the way training can. The thing that most often goes wrong is taking over too early: correct the policy before it has made its mistake and the recording holds your plan rather than its repair.

# Let the trained policy drive, take over with Tab, and record twenty corrections.
lerobot-rollout \
  --strategy.type=dagger \
  --strategy.num_episodes=20 \
  --policy.path=outputs/train/act_my_dataset/checkpoints/last/pretrained_model \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --teleop.type=so101_leader \
  --teleop.port=/dev/ttyACM1 \
  --dataset.repo_id=${HF_USER}/my_corrections \
  --dataset.single_task="Put the cup in the box"

LeRobot gives you the switching between the two drivers, the tagging of the frames you drove, and a new dataset in the same format as your original recordings. You supply the judgement about when to take over, and then a second training run, because the corrections are a dataset and not an update. By default only your correction windows are kept, each as one episode, and --strategy.record_autonomous=true keeps the policy's own frames as well.

6.5 SmolVLA, if fifty recordings are not enough#

Worth betting on, because it is the cheapest way to stop training from nothing. Size m, a laptop to run it and a big card to train it, Apache-2.0 for the code and for the weights.

SmolVLA was released by Hugging Face in June 2025. Book 3's page on foundation models records that it combines a vision-language backbone with a smaller action-producing part, was trained on about 10 million frames from 487 datasets contributed by the community, and reports about 78 per cent success on real SO-100 arm tasks. It is behaviour cloning, and what makes it different from ACT is that the weights already exist and that the policy also reads a sentence saying which task to do.

The one idea in SmolVLA is that nothing about reading a picture should be learned from your fifty recordings. ACT starts its picture part from a ResNet trained to name the objects in ordinary photographs, and everything else it knows comes from your recordings. SmolVLA starts from a vision-language model, which is a network trained on pictures paired with text, so it already connects what a cup looks like with the word cup. That network is kept, and a much smaller part called the action expert is attached to it to turn its output into moves.

Inside, the pictures, the sentence and the arm's own joint positions all become one sequence for that big network to read, where each picture is compressed into a small number of pieces and the joint positions become a single piece. The action expert is a transformer about three quarters as wide as the big one, and its layers alternate between looking at what the big network produced and looking at the other moves in the chunk it is building. Two published choices keep it affordable. The expert reads the big network only up to the middle of its layers rather than waiting for the last one, and at run time only the whole picture is used rather than several cropped tiles of it.

The chunk itself is produced in a way ACT does not use, and this is the part that matters for the problem of demonstrations that disagree. During training, noise is added to the recorded chunk, and the expert is asked for the correction that points from the noisy chunk back to the recorded one. At run time the chunk starts as pure noise, and that correction is applied in ten small steps until a chunk is left. This is called flow matching. Because the starting noise is drawn fresh each time, the network describes a whole spread of chunks that fit the picture and each run takes one of them, where ACT sets its 32 style numbers to zero and gives back the middle of the spread. So SmolVLA answers the averaging problem the way the diffusion policy two pages on does, and not the way ACT does. The pictures and the sentence pass through the big network once and the result is kept, so only the small expert runs the ten steps, which is what makes a model of this size usable on an arm at all.

What the pretrained weights buy is that your recordings adjust a policy instead of creating one, and what the sentence buys is one policy for several tasks. What the design costs is the size of that big network, which has to be loaded and run at every step even though only the small expert produces the moves. That is why this is the one row of the table that a laptop can run and should not train. On a robot arm the difference shows up when you have recorded thirty episodes rather than a hundred, and the ACT policy you trained on them does the right motion in the wrong place.

You would pick it over ACT in two cases. The first is that you start from trained weights rather than from nothing, which is the difference between fifty recordings being too few and being enough. The second is the sentence: one SmolVLA policy can be told at run time which of several tasks to perform, where an ACT policy trained on several tasks has no way to be told which one you want. Section 5 is about that input.

What it costs you is an extra install, because SmolVLA is not in the base LeRobot package, and attention to the sentence. The thing that most often goes wrong is that sentence, because the task text you pass when running must match the task text in your recordings. That text is an input to the network and not a label for you.

# SmolVLA is not in the base install.
pip install 'lerobot[smolvla]'

# Fine-tune the pretrained 450M policy on your own recording.
lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=${HF_USER}/my_dataset \
  --batch_size=64 \
  --steps=20000 \
  --output_dir=outputs/train/my_smolvla \
  --job_name=my_smolvla \
  --policy.device=cuda

# Run it, and tell it the task in the same words the recording used.
lerobot-rollout \
  --strategy.type=base \
  --policy.path=outputs/train/my_smolvla/checkpoints/last/pretrained_model \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --task="Put the cup in the box" \
  --duration=60

Hugging Face gives you a policy that has already seen 487 people's robots, so your recordings only have to teach it your room. You still supply the recordings, and its own documentation recommends about 50 episodes and reports that 25 was not enough. If training runs out of memory, check freeze_vision_encoder: it defaults to true, it holds the picture-reading part of the network fixed, and leaving it on is what keeps the memory figure above within reach.

6.6 How to choose#

The default is ACT in LeRobot. Record about 50 demonstrations on a cheap leader-and-follower arm, train ACT, run it, and watch where it fails, because going once around that loop teaches more than any further reading.

Four things change that choice. If you have no arm and want to compare plain behaviour cloning against a published number, use robomimic and its downloadable recordings, because LeRobot ships no plain behaviour cloning policy. If fifty recordings turn out not to be enough, or if one policy has to be told which of several tasks to do, fine-tune SmolVLA rather than recording hundreds more. If your ACT policy almost works and fails in the same place every time, do not record more demonstrations; run it with the DAgger strategy and correct it where it fails. And if your only machine is an Apple Silicon Mac, ACT is the one policy here that will finish, in about 6 to 14 hours rather than the hour a graphics card takes, with --policy.device=mps as the flag, which LeRobot's accelerator page documents. Training on the central processing unit alone is something its hardware guide tells you not to do.

ALVINN is on this page to be read and not to be run.

The two pages that follow this one are behaviour cloning as well. ACT and the diffusion policies copy demonstrations in exactly the way section 4 described, and they differ only in what they give back, which is what fixes the faults described below.

7. Where this is going#

This section is about the direction behaviour cloning is moving in. It is written on 4 October 2026, and it borrows the vocabulary that Book 3's frontier chapter uses for the four kinds of claim, because the difference between them decides how much weight a sentence can carry. A demonstration is a recording of something working once under conditions the publisher chose. A product announcement says a thing can be bought or downloaded, and you can go and check, which makes it the most useful kind. A research result is a measured number under a stated protocol. A projection is a statement about a date that has not arrived, and it is the weakest. Every claim below says which kind it is, and where the judgement is mine the sentence says so.

How it got here#

ALVINN put one network between a camera and a steering wheel in 1988, and for a long time the interesting part of behaviour cloning was the network. That has reversed. The network in ACT is small, ordinary and almost never the thing anybody changes, and the two things that decide whether a policy works are now the demonstrations you fed it and the weights you started from. Everything in the rest of this section is about one of those two, because that is where the work moved.

Where it is used in industry today#

Plain behaviour cloning is rarely a product. It is the method inside products, which means the honest industrial picture is thinner than the research picture and has to be read through what companies sell rather than what they publish.

The clearest commercial deployment claim belongs to Dyna Robotics, whose Dyna-2 model learns from recorded human behaviour and whose posts report a restaurant customer, Din Tai Fung, expanding from pilot sites to all its locations. Book 3 records that in its section on Dyna Robotics, and the honest label is a company claim rather than an audited figure, because no fleet size, no hours run and no intervention count has been published. Skild AI's S1 model is in commercial use with partners and cannot be obtained by you, and the company's own March 2026 partnership post says that ABB Robotics and Universal Robots plan to integrate its models, which is a plan rather than a shipped integration. Google DeepMind's Gemini Robotics 2 post offers its reasoning model through Google AI Studio and limits the two models that actually move a robot to early-access partners.

What is genuinely for sale, with a price on the vendor's own page, is the equipment for recording demonstrations. Trossen Robotics sells the Solo AI single-arm kit at $11,385.95 and the Stationary AI two-arm kit at $23,995.95, both descended from the ALOHA rig that the ACT paper introduced. It also sells TRumi, a handheld recording gripper, at $2,195.99 a pair with cameras, and Almond AI sells Mantis UMI at $1,799 a pair. At the other end of the ladder, the SO-101 bill of materials prices a leader-and-follower pair at $229.88. Those prices are checkable today, and they are the part of this field that has unambiguously become an industry.

The other measurable sign of use is the data. On 23 September 2026 Book 3 counted 77,598 datasets on the Hugging Face Hub carrying the LeRobot tag, most of them recorded by one person on one cheap arm. That is not an industry in the usual sense, and it is still the largest body of behaviour cloning data that exists.

What is being worked on right now#

Three threads dominate, and all three are about the supply of demonstrations rather than about the model.

The first is making a teleoperation rig cheap enough that the rig stops being the reason not to collect. The $229.88 figure above is the result, and 2026 spent its effort on what that rig cannot do. ArmnetBench, from July 2026, ran a farm of SO-101 cells, trained seven policies on twelve tasks with 50 demonstrations each, and released 2,518 policy rollouts labelled successful, suboptimal or failed. That research result is the first evidence of the cheap arm being used as evaluation equipment rather than as a hobby project. Phone2Act replaces the leader arm with a smartphone through Google ARCore. Neither is a product.

The second is collecting without a robot in the room. The Universal Manipulation Interface put the gripper in the operator's hand with a camera bolted to it, and its code and hardware files are MIT-licensed, so this one is downloadable. The important recent result is HiFi-UMI, from July 2026, which reports that a policy post-trained only on handheld demonstrations matched in-domain teleoperation on a real robot across three model backbones, with success differences of −2.5, +3.1 and −0.6 percentage points. Read its protocol before believing the headline. The comparison used 3,200 handheld trajectories per task against 300 teleoperated ones, so it compares two pipelines at their natural throughputs rather than one demonstration against one. The rig is proprietary and 2,000 hours of its data is downloadable under CC BY 4.0. YUBI, from Toyota's Frontier Research Center with AIRoA, published open handheld hardware under a licence permitting commercial manufacture, behind a corpus of 8,434 hours recorded by 179 operators at 22 desks. Book 3's handheld gripper section covers the whole family.

The third is the open policy library, and it is the thread that most changes what a beginner can do in an evening. LeRobot shipped version 0.6.0 on 6 July 2026 with depth cameras supported end to end, roughly twice as fast data loading, and a command that runs a vision-language model over your recordings and writes the task sentence for each episode. That last one removes one of the genuinely tedious parts of preparing a dataset. The same release added world models and vision-language-action models as downloadable policies, and NVIDIA put GR00T N1.7 and a teleoperation framework into the library on the same date. These are product announcements: you can install the version and fetch the weights.

Running underneath all three is the move from training a policy per task to fine-tuning one that was already trained. SmolVLA and GR00T N1.7 are downloadable, and openpi carries checkpoints for π0 and π0.5. Physical Intelligence has announced π*0.6 and π0.7 and released weights for neither, so the frontier of this line is a demonstration and the second tier is what you can have.

What is still unsolved#

The problem that has resisted the most work is that nobody has shown a reliable way to tell whether a policy will work before you run it on the arm. Training loss does not predict it, a held-out test set does not predict it, and section 2 of the evaluation page explains why. The research result that makes this concrete is LIBERO-VPro, a September 2026 study of six foundation models across 12 categories of visual disturbance, roughly 196,000 simulated episodes and 200 real trials on a Franka Research 3 arm. It found that the models tolerated whole objects being hidden and degraded sharply when the small local cues at the point of contact were disturbed, and that they were very sensitive to stale or missing camera frames. So a clean benchmark is not measuring the thing that will fail. Meanwhile real trials are slow and are almost never reproduced by anyone but the authors, which Book 3 summarises as evaluation costing more than collection.

Two more have not moved either. Every route that removes the robot from collection produces trajectories that nothing has verified the robot can execute, because the arm's reach and joint limits are unknown at recording time, and the filter that removes infeasible motions afterwards is a heuristic. And force is missing from almost every open dataset, because colour video and joint angles are cheap to record and contact force is not, so the policies in this chapter output positions and have nothing to say about how hard to push.

The quietest unsolved problem affects every number you will read. Two LeRobot datasets can share every field name and still differ in frame rate, camera placement and what counts as an episode, and nothing in the format records the difference.

The next two to three years#

The four expectations below are mine. None of them is an announcement, and I have given the reason for each, because a prediction without its reason is worth nothing.

I expect fine-tuning a pretrained policy to become the normal first move, and training from scratch to become the thing you do when fine-tuning has failed. The reason is arithmetic rather than fashion. Recording fifty more demonstrations costs a person an afternoon, and a fine-tune of a 450-million-parameter policy costs a few hours of one graphics card, so the cheaper move changes as soon as a downloadable checkpoint exists for your robot shape. Three such checkpoints already sit in one library with one training command. What would make me wrong is the open releases drying up, which is a real possibility: Book 3 notes that two of Google DeepMind's three 2026 robotics models went to early-access partners only.

I expect robot-free collection to become a normal way to get a first dataset, and not to replace teleoperation. The reason for the first half is that HiFi-UMI's result removes the last structural argument for owning a robot before you have a policy, and that assembled handheld rigs now have prices on vendor pages rather than being a printer and a weekend. The reason for the second half is the limitation above: nothing in a handheld recording knows your arm's kinematics, so somebody still has to run the policy on the arm, and the fastest way to fix a policy that fails in one place remains driving the arm yourself, which is what section 6.4 said about DAgger.

I expect the LeRobot dataset format to decide which methods get used, more than any measured quality difference between them. The reason is the 77,598 figure. A method that cannot read the format people publish in has an adoption problem that has nothing to do with whether it works, and formats with that much data behind them do not get replaced by better formats.

I expect evaluation to become the active research area, and I do not expect a reliable way to predict a policy's success rate before running it to arrive inside this window. The first half rests on a measured trend: Book 3's count of arXiv robotics abstracts shows the share mentioning a benchmark rising from 15.27 to 20.04 per cent between 2025 and 2026, faster than most of the methods, and ArmnetBench and the UMI Arena competition are what that looks like in practice. The second half is a judgement, and the reason is that predicting whether a grip will hold means knowing the contact forces, and the datasets do not contain them. You cannot predict from a signal nobody recorded. Until force appears in large recordings, the only honest way to find out whether a policy works will remain running it a hundred times and counting, which is what the evaluation page teaches you to do.

8. Where to read next#

The next page, action chunking transformers, shows how choosing a burst of moves at once reduces the adding-up of small mistakes. Then the page after it, diffusion and flow policies, shows how to keep two good ways of doing a task apart.

To compare copying with learning by trial, read reinforcement learning policies. And to go back to the list of all the kinds, read the chapter overview.

For more on where demonstrations come from, read where the data comes from, and for the rigs people use to record them, read Book 3's data and demonstration page. For a more critical account of behaviour cloning, with published numbers, read section 1 of learned methods for one arm. Its section on interactive imitation explains DAgger and its modern versions.