What is different

This chapter is about solution 6. It takes the borrowed model from solution 5 and trains it a little more on answers given in this cell. It is for a reader who knows the method from the pushing book's chapter on it. This page answers one question. What does that training buy, when the teacher it copies spends most of its time looking?

The short answer is that it bought work and not care. The trained model pushes more often than the untrained one. It also breaks more glasses. It separates no table.

Contents#

1. Introduction
2. A film of five tables
3. What it was trained on
4. The first fit learned to look
5. Two changes, and neither is a new model
6. The pair: work, but not care
7. The paths took the teacher's shape
8. Three times the steps moved the paths, not the table
9. The mirror image of the pushing problem
10. What the held-out queue measured

1. Introduction#

Fine-tuning means training a borrowed model further on your own examples. The pushing book explains it in fine-tuning — continuing somebody else's training. Here only a small correction is trained, by low-rank adaptation. Low-rank adaptation explains that method. The recipe is the pushing problem's, unchanged, so that the two problems' fits can be compared.

This solution and solution 5 form a matched pair. A matched pair is two runs that differ in one thing only. Here that thing is the training. Both run the same loop, the same stopping rule and the same clearance check. In code, this solution is solution 5's loop with other weights inside: fine_tuned.py builds on solution 5's as_downloaded.py.

What is new in this problem is the material the model learns from. The film below shows the result first.

2. A film of five tables#

The film shows the trained model as it is now. It uses tables 10001 to 10005, the same tables as solution 5's film, so the two can be watched side by side.

On the first table, 10001, the model gives 19 answers in 61 arm seconds. One of them, the third, comes down at a glass. That push does not give the glass room. The model is then taken to say "done", with the table as it found it. Solution 5's film of the same table has no push at all. The project's results page gives these counts.

So one push is the whole visible difference. The next section looks at what the model was trained on, because that explains why.

3. What it was trained on#

The film shows a model that rarely pushes. The cause starts with its training material.

The material is demonstrations. A demonstration is a pair of what a solution saw and what it then did. They come from solution 2, the teacher, run on 300 tables from table 1000. That gave 11,016 demonstrations, with 1,850 more kept back for checking. Of the 11,016, 8,965 are looks and 2,051 are pushes. So four answers in five only move the camera. Why the learned ones failed explains why this mixture harms every copying solution.

Three steps turn a demonstration into a target the model can learn.

• The glass readings are dropped. The model's state input holds only six numbers, for where the jaw is. So the glasses reach it only through the picture.
• The first drop of each push is removed. The cell lowers the jaw by itself, so asking for it would ask twice.
• Each path keeps the teacher's spacing. The cell takes waypoints at a fixed rate, so spacing sets speed. A short path is padded with its last pose. Of the 2,051 pushes, 1,824 fit in the model's 50 rows this way. The rest are squeezed and so run faster than the teacher's.

The project checked this reading on all 12,866 demonstrations. Each one was turned into the model's numbers and back with no measurable error.

The fit itself is small. 3,813,376 of the model's 453,859,552 numbers move. Each fit is 1,000 steps and takes 26.5 minutes on this laptop. Three fits were made, one per seed, because one fit is not a measurement. These figures are in weights/training.json and the solution's README.

The first fit drew its batches in the teacher's own proportions. The next section shows what that taught the model.

4. The first fit learned to look#

The last section found the material is mostly looks. The first fit copied that mixture as it came.

It learned the looking and little else. Its paths stayed high. Only about one answer in fifty came down far enough to move a glass. It also ran without the clearance check.

On the held-out queue it broke 8 glasses and separated no table. What the held-out queue measured gives its whole column.

So two things were changed. The next section says what they were.

5. Two changes, and neither is a new model#

The first fit was drowned by looks and carried out unsafe paths. Each change answers one of those faults.

Each batch is now drawn evenly from looks and pushes. The material itself is unchanged. Only the way a batch is drawn from it changed. Each fit now reads 4,000 demonstrations, about half looks and half pushes. The recipe is otherwise the same: the same steps, the same batch size and the same rate.

Every answer goes through the clearance check. This solution gets it from solution 5's loop. A path that cannot be shown to touch only the glass it names is raised to travel height. What fixed them and what did not describes the check.

Neither change touches the model or the recipe. The next section measures what the training does now, against the same weights untrained.

6. The pair: work, but not care#

The last section gave both halves of the pair the same mixture rule and the same check. So now the training is the only difference between them.

Both halves walked 20 tables below the held-out line, three times each. The tables hold 84 glasses. The table below gives the median of three walks, with the least and the most in brackets. The left column is the untrained model. The right column is the trained one.

The numbers are in pair-untrained.json and pair-fitted.json.

Training bought work. The trained model frees two glasses where the untrained one frees none. It also gives more answers.

Training did not buy care. The trained model breaks two glasses where the untrained one breaks none. A table that loses a glass earns nothing, so the trained model's share ends lower than the untrained one's.

Neither half separates a table. That is the plainest part of the result.

The pair shows what the training did to the score. The next section shows what it did to the paths themselves.

7. The paths took the teacher's shape#

The last section found more work but no better score. This section measures the paths, which is where the training shows most clearly.

The table compares the two halves on the same twenty tables. Each figure is the median over every answer, then the median of the three walks. The last two rows give the range over the three walks.

The figures are in pair-fitted-asking.json and its untrained partner beside it.

Read the first three rows together. The untrained answer is one long, fast arc across the whole glass zone. The trained answer is less than half as long, in steps a third as long. That is close to the length and speed of the teacher's own pushes. The trained model also comes low enough to push more often.

So the training taught the size of a motion. It did not teach where to put the hand down. The clearance check still raises most answers: 274 to 330 of the trained model's answers in a walk.

A fair question follows. Was the training simply too short? The next section answers it.

8. Three times the steps moved the paths, not the table#

The last section showed paths moving towards the teacher's. This section asks whether more training would carry that through to the table.

The project fitted the correction again. It used 3,000 steps instead of 1,000. It used solution 3's larger set, from 900 tables instead of 300. Each fit took about 80 minutes instead of 27.

The fit measures how far its paths are from the teacher's, every 500 steps. So one long fit shows what each change did. At step 1,000 the larger set had changed nothing. The extra steps then brought the paths closer. Take the median distance over all kinds of answer. After 1,000 steps it was 131 to 142 mm across the three fits. After 3,000 steps it was 83 to 126 mm.

The paths changed further. Their lowest point fell to 91 mm, and their travel to 330 mm. The table did not change. On the twenty tables it still broke 2 glasses, and its share was still 0.369. On the held-out queue it scored 0.345, against 0.405 for the shorter fit. So the longer fit was not kept. These numbers are in the solution's README.

So copying the teacher more closely is not the limit. The teacher mostly looks. A close copy of it is a solution that mostly looks. The next section sets this beside the pushing problem, where the same model failed the opposite way.

9. The mirror image of the pushing problem#

The last section showed that a closer copy did not help. Comparing with the pushing problem shows why the material, not the model, sets the result.

In the pushing problem the camera hung above the table. So every demonstration was a push. That fit learned the height of a push but not its length. What fine-tuning closes, and what it cannot touch describes it.

Here the camera rides on the wrist. So four answers in five are looks. With the even mixture, the fit learned the length and speed of a push. But its lowest point, 112 mm, sits between the push height and the travel height.

The table puts the two fits side by side. Each row is one property of the fit.

Neither fit learned where to put the jaw down. That is the one thing the task needs. The camera's position decides the mixture of answers. The mixture decides what the fit learns.

All of this was measured on tables below the held-out line. The next section gives the score on the held-out queue.

10. What the held-out queue measured#

The last sections measured the training on tables below the line. This section gives the score on the 50 held-out tables, with the 15 arm minutes from the examiner.

The table compares two walks of this solution. The left column is today's version. It gives the median of three walks, with the least and the most in brackets. The right column is the first fit, before the two changes.

The left column is from results.json. The right column is from the solution's README.

The gain is zero in all three walks. Every glass pickable at the end was already pickable at the start.

The two changes bought safety only. Broken glasses fell from 8 to none in two walks and one in the third. The share rose because no table was lost, so the glasses that started with room now count.

Every "done" was said while the table was still crowded. Those are sixteen answers in a row that the check raised. The rule rightly reads them as the model proposing nothing.

The pair showed +2 freed, and the queue shows none. Both numbers are small. Each comes from only three walks. The honest reading is that the training frees about nothing either way. On the queue the paths keep the shape the pair measured. Their lowest point is 109 to 134 mm and their travel is 325 to 358 mm, by asking.json.

The results set this solution beside the other five.

← A foundation model as it downloads — questions and answers · Questions and answers →