What is different

This chapter is about solution 5. It runs SmolVLA, a large borrowed robot model, exactly as it downloads. Nothing in it is trained in this cell. It is for a reader who has read the earlier chapters of this book. It also assumes you know the model from the pushing book's chapter on it. This page answers one question. What happens when the same untrained model gets the camera on the wrist and an open run?

The short answer is surprising. The wrist camera is the view the model was trained with. Yet on its first walk the model broke twenty glasses. This page says why, and what made it harmless again.

Contents#

1. Introduction
2. A film of five tables
3. What stays the same
4. The first walk broke twenty glasses
5. The wrist camera did not bring the answers down
6. Aiming at nothing is worse than not arriving
7. The clearance check made it harmless
8. What the held-out queue measured

1. Introduction#

In the pushing problem, this model saw a picture from a camera above the table. Its answers stayed high in the air. It broke little and did little. What was expected, and what happened gives those numbers.

This problem takes the overhead camera away. The arm carries its own camera on the wrist. Most real robots in the model's training data had a camera there. So the input is now closer to what the model knows. The project wrote down a prediction before the run. The answers should look more like pushes.

The answers did come down. But they came down into the glasses. The rest of this page explains that result. The film below shows the model as it is now, after the repairs.

2. A film of five tables#

The film shows what the model does on five tables today, so it comes before the history. Every solution in this book is filmed on the same five tables, 10001 to 10005.

On the first table, 10001, the model gives 16 answers in 71 arm seconds. None of them is a push. The arm crosses the table again and again, well above the glasses. At the end the model is taken to say "done", with two of the four glasses still crowded. The comparison table in the project's results page gives these counts.

The film shows a harmless arm. The next section says which parts of the solution did not change to get there.

3. What stays the same#

Before the changes, it helps to know what was kept. The model is the same published checkpoint as in the pushing problem. The code that turns its output into jaw waypoints is imported from there, unchanged. So is the code that loads and asks the model. What is borrowed, and what is not describes those parts.

Two pieces are new, and both are shared with other solutions. The model has no way to say "done". So it uses the shared stopping rule from looking, remembering and stopping. It also runs the shared clearance check on its own answers. What fixed them and what did not explains that check. The solution's own loop is in as_downloaded.py.

The clearance check came later. The first walk of the held-out queue ran without it, and the next section shows what that walk did.

4. The first walk broke twenty glasses#

The parts in the last section were the same on both walks, except the check. This section reports the walk without it.

The first walk broke 20 glasses. What the held-out queue measured gives its other columns. It never once said that a table was finished. Each table ended when a glass fell or the clock ran out.

Why did it never say "done"? The shared rule waits for a run of answers that cannot touch a glass. Almost every answer of this model came low enough to touch one. So the run never formed. The rule was right. A hand at that height really is among the glasses.

So the answers came low. The next section asks what made them come low, because the obvious cause turned out to be wrong.

5. The wrist camera did not bring the answers down#

The last section found answers low among the glasses. This section separates the two things that changed between the two problems.

The first change is the picture. The model now sees the wrist view.

The second change is the jaw pose the model is told. A model of this kind is told where its hand is, as well as shown a picture. In the pushing problem, the jaw was parked high between pushes. So the pose it was told was always at the top of its range. Here the model is told the last waypoint of the path the arm really carried out. That is a real pose near the table. Nobody chose this for the model. It follows from this problem having no separate "look" step.

The project ran a check to tell the two apart. It walked twenty tables with the wrist picture but with the old parked pose. So only the picture differed from the pushing problem.

The table compares three runs. Each column is the median answer. The first column is from the pushing problem. The next two are this problem's twenty tables below the held-out line.

Read it from right to left. With the real pose, the answers come down to 145 mm. With the parked pose and the same wrist picture, they stay at 236 mm. So the drop comes from the pose, and the picture adds none of it. The middle and right columns are the fair comparison: same tables, same code, one change. The left column uses other tables, so 236 against 217 means "no lower" and no more.

The cause is how the model works. It answers with targets near the pose it is told. Tell it the hand is high, and it answers high. Tell it the hand is low, and it answers low. The picture does not change this.

The figures come from pinned-asking.json and measurement-asking.json. The parked-pose check is one walk, not three. The left column is from the pushing book.

So the answers now reach the glasses. The next section asks why that made the model worse rather than better.

6. Aiming at nothing is worse than not arriving#

The last section showed the answers now reach the table. This section explains why reaching it did harm.

The model still does not know what is in the picture. That is the domain gap, which the pushing book explains. So its paths do not aim at any glass. A middle answer travels 833 mm. The glass zone is only 320 mm by 360 mm. So one answer crosses the whole zone.

The middle answer comes down to 145 mm, which is the middle of the jaw. The fingers hang about 70 mm lower, so they sweep the tops of the glasses and tip them. Hands that swept the rims explains this fault.

In the pushing problem the model was wrong in a harmless direction. Its hand moved through empty air. Here it is wrong by about the same amount. But now the error lands on the glasses. A path that reaches the glasses without aiming at one does more harm than a path that never reaches them. Moving a model's input closer to its training data does not make its error harmless.

The cell's own safety interlock did not stop this, for the reason the one safety check the cell keeps gives. So the solution needed its own check, and the next section shows what that check did.

7. The clearance check made it harmless#

The last section showed paths that sweep the glasses. This section shows what the shared clearance check did to those paths. The weights did not change.

The check raises to travel height any path that is neither a high look nor a low push at one named glass. The clearance check explains it in full.

Over three walks, the check raised 658 of this model's answers. That is every answer it could not accept. The table lists each reason the check gave. Each row counts the answers raised for that reason.

The counts are in results-raised.json. The top reason says the most. The model comes down to about the right height. But it does not come down at a glass.

A raised answer touches nothing. So the run of quiet answers now forms, and the stopping rule ends the table. The model is idle, and now the rule can see that. The next section shows what this did to the score.

8. What the held-out queue measured#

The last section said the check raises nearly every answer. This section shows the score that follows. The queue is the 50 held-out tables and the 15 arm minutes from the examiner.

The table puts the two walks side by side. The right column is the walk without the check. The middle column is the walk with it. That column gives the median of three walks, with the least and the most in brackets. The weights are the same in both columns.

The middle column comes from results.json. The right column is from the table in the solution's README.

Twenty broken glasses became none. Nothing was retrained. The check alone did it.

The share rose, but the model did almost nothing. Once no table is lost, the glasses that started with room count. Those are 16 of the 18 pickable glasses at the end. So the gain of +2 is the honest number, as the share needs a baseline explains.

Two walks of three separated one table. Each of those walks freed two glasses. The third walk freed none. So this is luck, not skill. A model that never saw this cell will sometimes shove a glass the right way.

It reaches fewer tables now. A raised answer still costs the arm its move. The stopping rule needs 16 quiet answers in a row. So a table costs about 19 answers and about 80 arm seconds. Before, a falling glass ended a table sooner. The two separated tables took 123 and 160 arm seconds.

The thinking time fell from 776 ms to 280 ms. That is not the model. The first walk ran while the laptop was also training other solutions.

The clearest finding is about the camera. The wrist view did not help this model at all. The same model fine-tuned here asks whether training in this cell can help, and this solution is its baseline.

← A world model, then plan with it — questions and answers · Questions and answers →