What it uses. It uses LeRobot, a library of robot learning code built on PyTorch. It also uses one borrowed model, SmolVLA, exactly as it downloads. No number in the model is fitted in this cell. What it does. It shows the model a top-down picture of the table, one fixed sentence, and the jaw's pose. It then carries out the waypoints the model returns. Then it looks again and asks again. This repeats until every glass has room or the push budget is spent. How the output is produced. The model returns 50 actions. These are read as waypoints for the jaw. The examiner moves the jaw along them directly, without its push macro. What it costs. It needs no demonstrations, no labels and no training. It needs the library, a 2.9 GB download and a laptop. The remaining cost is time, because every push needs a pass through a large network.
The cell is described once, in the cell. That page covers the layout, the two camera positions, the four sensors, and the words this project uses for them. This page covers only what is specific to this solution.
Contents#
1. Introduction#
This chapter answers the problem this book sets in the simplest possible way. It downloads a general robot model and runs it. Nothing is collected, trained or fitted in this cell. The model is shown the table, told in words what to do, and asked for actions. Whatever it returns is carried out.
This solution is built and has run. The code is in
05-smolvla-as-it-downloads/.
It runs on the same 50 held-out tables as the other solutions. It did
badly, and in an unexpected way: most of its pushes pass high above the glasses
and touch nothing. How it
works gives the
measured details.
There are two reasons to build it. First, it is the cheapest solution in this book to try, because the whole setup is a download. Second, it is the starting point of the cleanest comparison in the book. Solution 6 is this same model with more training done here. So the gap between the two measures what that training gave, and nothing else. This means the score of this solution matters less than the comparison it makes possible.
This page says what kind of model this is, what goes in and what comes out. The pages after it explain where the model's skill comes from, what it cannot use, and why a poor result here is still useful.
The introduction said what this solution is. The next section shows it at work on one table.
2. A film of one table#
The introduction described the borrowed model. This film shows it on one table in the simulator, at one and a half times real speed.
The table holds six glasses, where the average is five. Five of them have no room at the start.
In the film, the jaw sweeps across the table sixteen times. It never reaches a glass, because the model's answers stay high above the table. One glass is racked, and only because it already had room. This is the longest of the six films and the emptiest. That is the real result, not a fault in the recording.
The model draws a fresh answer each time, so the film is one draw, not an average. The scored runs on this table racked one glass each and made thirteen pushes on average. So the film is a little busier than a typical run.
The six measured solutions are all filmed on this same table. The six films collects them and says how the table was chosen.
The film shows a capable model acting in the wrong place. The next section says which problem an unchanged model is meant to answer.
3. The problem this solves#
The film showed what the model does. This section says why anyone would try an unchanged model at all.
The problem asks for jaw movements, one after another, until every glass has about 76 mm of clear room and nothing has fallen over. This is hard because a push does not go where it was aimed. Pushing without toppling explains why.
Every other solution in this book builds something for this. Solution 1 builds a rule. Solution 2 builds geometry and a ranker. Solution 3 learns a policy from recorded pushes. Solution 4 learns a model of the table and searches through it. Each of these costs work, and three of them need data collected in this cell.
This solution does something else. Pushing objects on a table is not special to this project. Many people have recorded robots doing this kind of thing. Someone has already trained one model on a huge pool of those recordings. So this solution borrows a model made for tables in general. It asks if that is good enough for this table. The answer is worth knowing before anyone spends a week recording demonstrations.
The question is now clear. The next section gives the idea that answers it.
4. The main idea#
The last section said this solution borrows instead of builds. This section gives the three steps that make up the whole idea.
First, download the model and show it the table from the top. The test examiner draws this picture so that solutions that read pictures get the same information as solutions that read numbers. A fixed camera 750 mm above the glass zone looks straight down. It gives a colour picture of 384 by 384 pixels.
Second, tell it in words what to do, and give it a pose. The instruction is the same every time: the glasses are too close together, push them apart. The pose goes in the slot the model keeps for an arm's joint readings. This cell puts the jaw's position there, because the model was trained on a different arm. This cell's joint angles would mean nothing to it. This small change turns out to matter a lot. How it works explains why.
Third, carry out what comes back. The model returns 50 actions. Read as waypoints for the jaw, they are already the output the examiner accepts. The examiner moves the jaw along them, and the table changes. The model is then asked again about the new table.
So this solution has no fitted numbers, no hand-written geometry and no search. It has a download, a sentence and a loop. Keep this in mind on the pages that follow, because almost every strength and weakness comes from it.
← A world model, then plan with it — questions and answers · The code →