What the repairs bought

This page answers one question. What did the two-model copying solution achieve, and what holds it back? The previous page described the gate, the policy and their inputs. This page measures them. It reads the held-out queue first, then the extra examples, then the glasses it broke.

Contents#

1. The held-out queue
2. Three fits, three very different walks
3. Why the share is lower than when it did nothing
4. Three times the examples
5. What topples a glass here
6. The push direction is often blocked
7. Why the direction was not handed to arithmetic

1. The held-out queue#

The previous page ended with a fitted solution. This section gives its score on the held-out queue. The examiner explains the queue and every column.

Each of the three walks used one of the three fitted copies. The table below gives each walk and then the median, which is the middle of the three. Read each row as one column of the card.

The numbers are in results.json.

The main change is that the solution now acts. The first version handed back every table as it was drawn. Now every copy pushes, and one copy separates two tables. Every copy also breaks exactly one glass.

The three columns disagree a great deal, and the next section asks why.

2. Three fits, three very different walks#

The table shows three copies of one recipe giving very different walks. This section explains the widest gap.

Copy 0 attempted 36 tables. The other two attempted 9 and 14. Copy 0 did not work faster. It gave up sooner. It said 33 of its 36 tables were done while they were still crowded. So it reached far down the queue by doing almost nothing on each table.

So "tables attempted" alone says nothing about skill. Three columns that mislead shows why.

Copy 0 also had 14 answers refused by the cell's interlock. The other copies had none.

The share is the number that looks worst, and the next section asks why it fell.

3. Why the share is lower than when it did nothing#

The walks differ, but all three end with a share near 0.36. This section explains why that is below the share of a solution that did nothing.

The first version had a share of 0.400. It moved nothing. So 0.400 was simply the glasses that arrived with room. The project's page for this solution gives that number.

A solution that tries can also lose. When a glass falls, its table earns nothing, as a topple loses the table explains. So a share falls as soon as a solution starts to try. Here the breakage is what holds the share down.

The fall is also inside the noise. The solution's README compares six walks on each side. The version fitted on 300 tables had a share of 0.400, from 0.316 to 0.404. The version fitted on 900 tables had 0.361, from 0.303 to 0.423. The difference is about one and a half standard errors. That is too small to call a real change.

So read the gain row and the broken row together. The share alone does not describe this solution.

That comparison raises the next question. What did three times the examples change?

4. Three times the examples#

The previous section compared versions fitted on 300 and 900 tables. This section says why the examples were tripled, and what that bought.

A fitted model can be measured against its teacher twice. Once on the tables it was fitted on, and once on tables it never saw. If the first distance is much smaller, the model has memorised its tables rather than learned the task. More examples are the cure for memorising.

The table below gives the policy's distance from the teacher, in millimetres for each waypoint. Read each row as one measure, and each column as one version.

The 900-table numbers are in weights/training.json. The 300-table numbers are in the solution's README.

At 300 tables the two distances were far apart, so the policy was memorising. At 900 tables the gap is smaller. The error on fitted tables rose by about 3 mm. The error on unseen tables fell by about 1 mm. The direction of a push came about 6° closer to the teacher's.

On the table, broken glasses halved. Over six walks a side they went from 2 to

1. Nothing else moved outside the noise. The gate did not improve at all. Its readings spread wider on three times the material.

So the reading was right, but the cure was small. What limits this solution is the model and the task, not the number of examples.

Breakage was the one thing that clearly changed. The next section asks what breaks a glass now.

5. What topples a glass here#

More examples halved the broken glasses. This section looks at the five glasses that still fell.

A solution cannot see its own breakages. A table ends the moment a glass falls, so the next look is never taken. So the measuring script, run.py, keeps each table's last answer beside how that table ended.

It was run on tables 0 to 19, with four arm minutes for each table. The three copies lost five glasses there. All five followed the same kind of answer. Each was a push at a named glass. Each passed the whole clearance check. Each was then carried on by arithmetic until the glass would have room. What fixed them and what did not explains the clearance check and the carrying on.

Four of the five pushes were expected to move their glass 32 to 37 mm. That is about half the room a glass needs. So these were real attempts at the task, not small shoves. The numbers are in measurement-fits.json.

The pattern is this. The solution now makes fewer pushes, but longer and better aimed. The better it aims, the further the arithmetic carries its push. And a longer push carries more risk.

Nothing the solution can compute would have refused these pushes. The glasses had been watched sliding, and no rule predicts a slide for any glass in this cell. So arithmetic cannot help here. What is left is the model that draws the push.

Breakage is one limit, and the next section names the commonest one.

6. The push direction is often blocked#

The previous section found the limit of arithmetic. This section shows where the policy itself falls short most often.

On tables 0 to 19, the clearance check raised 71 to 150 answers in each copy for one reason. The path would take the hand through a glass it was not aimed at. A raised path goes to travel height and moves nothing. This is the commonest reason an answer is refused.

The policy draws the teacher's push shape, in about the right place. But it draws it in a direction the table does not allow. Three times the examples brought that direction 6° closer. It was still the commonest refusal.

Two other limits are shared with the other learned solutions. One push moves a glass only 15 to 32 mm here, and a glass needs about 76 mm. And every push needs a probe first, because no glass can be predicted to slide. Why the learned ones failed explains both.

The blocked direction has an obvious repair. The next section asks why it was not made.

7. Why the direction was not handed to arithmetic#

The previous section showed that the direction is the main weakness. This section explains why the repair is left out on purpose.

The geometry solutions already choose a push direction from the measured neighbours. The same arithmetic could choose the direction here. The policy would then draw only the shape of the push. That would very likely help.

But this solution exists to measure one thing. What does copying a teacher achieve when looking is most of the task? With an arithmetic direction, it would become the geometry solution with a learned chooser in front. That may be a better machine. But it answers a different question, so it should be built as a new solution.

A gentler version was tried. When the arithmetic refused the gate's first glass, the solution asked the gate for its next choice. The model still chose the order, and arithmetic could only refuse. It broke more glasses than before. It changed only about one answer in twenty. Those were answers at glasses the gate itself had ranked low, where it is often wrong. What changed in the project gives the numbers.

That ends this solution's results. The questions and answers page asks the short questions a reader may still have.

← What is different · Questions and answers →