What the repairs bought

This page gives the world model's score on the held-out queue, and the repairs that led to it. It also gives the project's newest finding: which glass falls when this solution breaks one. The score comes from results.json. The other numbers come from the solution's README.md, unless a line names another file.

Contents#

1. The score on the held-out queue
2. Why the share is lower although it separates more
3. The first repair: a search that distrusts its model
4. The second repair: twice the pushes, and a heavier charge for a missed topple
5. The topple filter on the search's own pushes
6. Which glass goes over

1. The score on the held-out queue#

The previous page showed what the port changed. This section shows what the result was.

The queue and its columns are described in the examiner. The table below gives the median of three walks, with the least and the most in brackets.

It separates more tables than any other solution in this book. But its share is below the geometry reference's 0.627. The card of six sets it beside all the others.

The share and the separations point opposite ways, so the next section asks why the share is lower when the separations are higher.

2. Why the share is lower although it separates more#

The card above has two rows that point opposite ways. This section explains that with one fair comparison and one rule of the examiner.

The fair comparison cuts both walks to the same tables. On the first 26 tables, which both reached, the world model and the reference each separate 10. The same tables for every row gives both rows in full. So the world model does not win on any one table. It wins on the clock, by reaching more tables.

The rule is that a table with a broken glass earns nothing, though its glasses stay in the total. The broken glasses cost the world model about 3 points of share. The rest of the 7-point gap to the reference is the tables it reached and left crowded.

The next section asks how the broken glasses were first brought down.

3. The first repair: a search that distrusts its model#

The broken glasses cost about 3 points of share, as the last section showed. This section gives the first change that reduced them.

The first walk of the queue broke 8 glasses and separated 11 tables. The reason is the winner's curse. A search that takes the best of many uncertain guesses picks the guess with the largest error. The winner's curse explains it in full. Nothing in the model was changed. Three things in the search were changed, and the previous page shows them in place:

• every reading is the worst of the three copies, not their average;
• a chance of a topple now has a price, as well as a limit;
• a push whose hand would sweep a neighbour is dropped before it is ranked.

A limit alone makes every push under it equally good. So the search used to pick the push at the very edge of the limit. The price makes it prefer the safe middle instead. After the three changes, the queue broke 1 glass in a walk. Separated tables went from 11 to 14.

The next section asks what happened when the model was then given more data.

4. The second repair: twice the pushes, and a heavier charge for a missed topple#

The first repair changed only the search. This section gives the one repair that changed the model, and the surprise it brought.

A second round of random pushes was collected. It took the fit from 6,886 pushes to 12,253. More pushes made the model aim better. But they also made it raise the alarm less often. The search's limit is fixed, so a calmer model lets more risky pushes through.

The fix was to make the fit pay ten times as much for a topple it fails to predict. The table below shows the 658 validation pushes. Read each column as one fit. The figures are from training.json and the README.

The middle column aims best and catches the fewest topples. The right column keeps most of the better aim and catches nearly as many topples as before. Neither change helps on its own, so the two are kept as a pair.

Below the held-out line, the pair separates 15.7 tables a walk against 14.3. It loses 1.7 glasses a walk against 3.0. On the held-out queue it breaks 5 glasses where the earlier fit broke 1. The pair was kept anyway. Where choices are made says why they are made below the held-out line, and not on it.

The next section asks why the topple filter misses so many of the pushes that break a glass.

5. The topple filter on the search's own pushes#

The last section measured the filter on random pushes. This section measures it on the pushes the search really makes.

A second check set was collected on 150 fresh tables. This time the search chose every push. The limit stayed at 1 in 100. The filter caught four topples in five among random pushes. It caught only one in three among the search's own. The winner's curse gives the counts and explains the cause. So no limit or price on the model's number could stop these pushes. What fixed them and what did not lists the repairs that were tried here and dropped. One of them refitted the model on the search's own pushes. It caught 22 of 40, but it broke more glasses, not fewer.

The next section asks where the remaining fault is: in the model, or in the arithmetic that keeps the hand clear.

6. Which glass goes over#

The last section showed the model misses most topples on its own pushes. This section checks which glass those topples are.

Two faults end a table the same way. If a neighbour goes over, the hand passed too close to it. Then the clearance arithmetic is at fault. If the pushed glass goes over, it tipped where it should have slid. Then the model's idea of toppling is at fault. These need opposite repairs.

The run's own record cannot tell them apart, because the table ends before the next look. So the project asked the physics engine directly, with make toppled in measure.py. That is allowed, because it only diagnoses and decides no push. It ran 300 tables from table 5300, below the held-out line.

The table below gives the answer. Read it as a count of the tables that ended with a glass over.

Every glass that went over was the pushed glass. Not one was a neighbour. So the clearance arithmetic did its whole job over 300 tables. The model's own chance of a topple for those 30 pushes was about 2 in 10,000 at the median. At the worst it was 8 in 1,000. The limit is 1 in 100. So the search never took a push it knew to be risky.

What is left is one fault. A glass tips instead of sliding, under fingers the model was sure about. The slide or tip limit depends on friction, and no camera here can see friction. So the next step is not a better learned chance. It is a geometric check of toppling, built from the contact height and the glass's measured foot. What to try next lists it with the other next steps.

The next page asks the short questions a reader may still have about this solution.

← What is different · Questions and answers →