The examiner is the shared code that draws the tables, runs the physics and marks each run. Why there is one examiner is the same as in the pushing book. This page covers only how the marking changed once nothing counts the pushes. It answers these questions. What is the budget, and what does it count? Which tables count as attempted? How can a table end? And what does each column of a result mean?
Contents#
- One budget for a queue of tables
- Arm seconds, not laptop seconds
- What "tables attempted" counts
- How a table ends
- A topple loses the table, not the queue
- The share needs a baseline
- The one safety check the cell keeps
- Where choices are made
- The columns of a result
1. One budget for a queue of tables#
Nothing counts pushes any more, so something else must stop a solution that never stops itself. This section says what that is.
Every solution gets the same queue: 50 tables in a fixed order, starting at
table 10,000. It gets 15 minutes of arm time for the whole queue, not for each
table. The queue is walked 3 times. These numbers are in
bench/judging.py
and
bench/open_run.py.
Each table may spend as much of the time left as it needs. So a slow solution reaches fewer tables. This makes speed part of the score, not a separate column.
The 15 minutes were measured, not guessed. On tables below 10,000, the simplest solution spent about 43 arm seconds on a table. So 15 minutes carries it through about 21 tables. The queue of 50 is longer than any solution can finish. A solution must run out of time, not out of tables, or the number of tables it reached would say nothing about its speed.
Table 10,000 starts the held-out tables, as in the pushing book. But this problem draws its own tables, so its table 10,000 is not the pushing book's table 10,000.
The queue is walked in order. So two solutions are only comparable on the tables they both reached, which is the start of the list. The results compare them that way.
The budget is counted in arm seconds, so the next section asks how a path is turned into seconds.
2. Arm seconds, not laptop seconds#
The budget is time. This section says whose time, and how it is added up.
The budget is simulated seconds of the arm moving. It is added up leg by leg,
in bench/clock.py.
Each leg of a path costs the longer of two times:
- one tick of the arm's controller, which is 0.05 s;
- the time to cover the leg at the jaw's top speed, which is 0.20 m/s.
Both values are in the pushing problem's
bench/bench.py.
So waypoints placed closer together make a motion slower, not faster. Nothing
can go faster than the top speed.
Three more rules make the count honest.
- The move between answers is charged. The simulator jumps the jaw to the start of the next path without moving it. A real arm would spend time on that move, so the clock charges the flat distance at top speed. Without this rule, a look from the far corner would cost the same as a look from where the jaw already is.
- Only real motion is charged. A path that stops early, because the jaw met a glass, costs only the part that happened.
- The lift home is charged. When a path ends, the cell lifts the jaw back to travel height. That is real motion too.
Why arm seconds? Two solutions that make the same motions keep a real cell busy for the same time, on any computer. Laptop seconds would compare computers rather than methods.
Thinking time is still recorded, in its own column. It tells you whether a method could run on a real arm at all. But it is not taken from the budget, because the arm is not the thing that was slow.
The budget can run out in the middle of the queue, so the next section asks which tables count as attempted.
3. What "tables attempted" counts#
The budget belongs to the queue, so near its end the clock holds only a few seconds. This section says what happens to the tables there.
A table can be handed over and asked for a path. The path may not fit in the time left. Then the arm does not move, and the table comes back untouched. Such a table is not counted at all.
Counting it would say two untrue things. It would say the solution did work on that table. And it would count every glass on that table as left crowded. That would pull the share down for a reason that has nothing to do with the method.
The walk then goes on to the next table, because a cheaper path may still fit.
So "tables attempted" is the number of tables the arm worked on. It is not how
far down the list the solution got. For example, one walk of the fine-tuned
model attempted 11 tables, and the last of them was table 10,016, the 17th in
the list. The rows are in
06-smolvla-fine-tuned/results-tables.json.
A table the arm did work on then ends in one of a few ways, so the next section asks what those are.
4. How a table ends#
An attempted table has to stop somewhere. This section gives the two labels a table gets: why the run stopped, and what state it left.
A table stops in one of three ways:
- said done: the solution answered "done";
- ran out: the next path did not fit in the time left;
- glass over: a glass went over, and the table stopped at once.
Separately, the examiner marks the state of the table at the end. Read the table below one row at a time. The left column names the state, and the right column says what it takes.
| state | what it takes |
|---|---|
| separated | every glass has room, and nothing fell |
| failed | a glass went over |
| unfinished | nothing fell, but a glass is still crowded |
The pushing book's endings were for racked and refused glasses. These three are for a table that stays on the table.
"Unfinished" covers two different faults. The clock may have stopped the solution, which means it was too slow. Or the solution may have said "done" too early, which means its judgement was wrong. The column "said done while crowded" counts the second kind, because the two need different fixes.
The marking reads where the glasses really are in the simulator. It never reads what the solution reported.
"Failed" is kept apart from "unfinished", so the next section asks what a topple costs.
5. A topple loses the table, not the queue#
A failed table is the worst ending. This section says how much it costs.
When a glass goes over, that table ends and earns nothing. None of its glasses
counts as pickable, even the ones with room. But every glass on it still counts
in the total that the share is taken from. This rule is in
bench/scoring.py.
So breaking a glass is never cheaper than leaving the table alone.
There is no free breakage. The run does not forgive the first topple. If it did, a solution could gamble at no cost. That would undo the rule that a refused glass is a result.
The run carries on to the next table. It does not stop at the first topple. Stopping would let one bad moment wipe out every table after it. The same method run twice would then give very different scores, and the noise would be larger than the gap between two methods.
Topples aside, the share counts glasses with room. The next section asks why a share alone can mislead.
6. The share needs a baseline#
The share is the partial credit. This section shows why it must be read beside the number of glasses freed.
The pickable share is the glasses with room at the end, out of every glass on every attempted table. But every table starts with one or two glasses that already have room. So a solution that does nothing scores above zero.
Here is the simplest solution's walk, from
01-one-fixed-nudge/results.json.
Its 26 tables held 102 glasses. At the start, 39 of them had room. At the end,
64 had room. So its share is 0.627, and it freed 25 glasses. Doing nothing
on the same tables would have scored about 0.38.
So each result gives the glasses freed beside the share. The glasses freed are the part the solution is responsible for. The results show the other columns that mislead when read alone.
The share counts what the solution did. The next section asks what the cell still does on the solution's behalf.
7. The one safety check the cell keeps#
The last sections described what the examiner counts. This section describes the one thing the cell still refuses to do.
Some paths are rejected because the arm cannot follow them. Examples are a path below the lowest height the hand can reach, above travel height, off the table, or too long. These are limits of the machine, not marks.
One real judgement is kept. The cell will not drive the jaw into a glass that would tip rather than slide. It asks the pushing book's slide-or-tip rule, with the range of friction anyone could work out, not the real friction. A refused path does not move the arm and costs no time. The cell counts each one. A real cell keeps its safety interlocks, so this one stays.
An interlock is a safety check that blocks one dangerous motion and takes no
other decision away. For all six solutions, the middle value over the 3 walks
is 0 refusals. The most in any one walk was 14, for the imitation policy. The
counts are in each solution's results.json, such as
03-imitation-from-demonstrations/results.json.
Yet glasses still fell in the first walks. They fell because the hand met them high up, while it was only travelling. The interlock asks about a push at the working height, so it had nothing to say about that. Looking, remembering and stopping explains where the hand really is, and what fixed them describes the check the solutions added for themselves.
A refused path costs no arm time, so a solution could repeat one forever. The loop also caps the number of answers on a table for that reason. The cap is set from the time left, so it never limits a solution that is spending its time.
The rules are fixed, so the next section asks where a solution may be tuned against them.
8. Where choices are made#
The rules mark a solution. This section says on which tables its settings may be chosen.
Tables below 10,000 are for building and choosing. The tables from 10,000 up are measured once, as in the pushing book.
What is new is that the tables used for choosing must have the clock too. A test with no clock gives every table as much time as it wants. It asks whether a table can be separated. The score asks how much gets done before the clock stops. A change that trades time for care, such as looking more or pushing in smaller steps, wins the first question and loses the second.
This happened. The project's record of its repairs says two settings of the world model were chosen with no clock. Both did worse on the scored queue than the version they replaced. So choices are now made on a clocked queue of 50 tables below 10,000, walked 3 times, from three different starting tables.
Each result gives a middle value and a range over the 3 walks. How a result is claimed explains why a number without a spread is not a result.
With the rules in place, the last section lists what a result file reports.
9. The columns of a result#
The sections above each added a count. This section puts them in one list.
Each solution's walks are written to its own results.json. Read the table
below one row at a time. The left column is the name used in this book. The
middle column says what it counts. The right column is the key in the file.
| column | what it counts | key in results.json |
|---|---|---|
| tables attempted | tables the arm worked on | tables_attempted |
| tables separated | every glass has room, nothing fell | outcome.separated |
| failed by a topple | tables that lost a glass | outcome.failed |
| glasses | every glass on the attempted tables | glasses |
| glasses freed | glasses with room at the end, minus at the start | glasses_end.gain |
| pickable share | glasses with room at the end, out of all glasses | glasses_end.pickable_share |
| glasses broken | glasses that went over | glasses_end.broken |
| said done while crowded | "done" on a table that was still crowded | said_done_while_crowded |
| interlocks | paths the cell refused | interlocks |
| arm seconds for a separated table | the average cost of a win | arm_seconds.separated_mean |
| thinking an answer | computer time per answer | thinking_seconds_per_answer |
Glasses freed can be negative, because a solution may push a glass into its neighbour. The arm seconds are averaged over separated tables only. A solution that gives up early would otherwise look cheap.
These are the rules that every chapter of this book is marked by. The next page asks the short questions a reader may still have about them.
← What changed: two helps taken away · Questions and answers →