Robotics Docs

Seeing the Glasses

One perception problem, answered six ways and measured

Several glasses of one kind stand on a table, and the arm has to work out which pixels belong to which glass. This book follows that single question the whole way down: what the camera can and cannot see, why a glass standing behind another is in no picture at all, and six quite different ways to answer it. One is a written rule with nothing fitted to anything, one is a small network trained in this cell from nothing, and four start from weights somebody else fitted to photographs of the everyday world. All six are built, and all six are scored by the same examiner, so the differences between them are measured rather than argued. The programs are in the repository, and each solution names the folder it was built in.

12 chapters47 sections5 h 30 min of reading
0 of 47 sections read
Start reading
  1. 01

    The Cell Everything Happens In

    Every solution in this book works in the same room. It uses the same arm, the same camera and the same table. So does every solution in pushing the glasses apart. This page gives those shared numbers once, so other pages do not…

    1 section · 18 min

    1. 1.1The cell — the layout, the sensors, and the words18 min
  2. 02

    The Problem

    This page states the task that every solution in this book must solve. It is for any reader, and you should read it before any solution.

    2 sections · 19 min

    1. 2.1What is asked for — segment the glasses10 min
    2. 2.2Looking again at what was hidden9 min
  3. 03

    The Examiner

    This book solves one problem in six ways. Six answers can only be compared if each one gets the same question and is marked the same way. The examiner is the program that does both. It works in four steps. It arranges glasses on…

    4 sections · 28 min

    1. 3.1The examiner — the same question for every answer10 min
    2. 3.2An example of the input5 min
    3. 3.3An example of the output4 min
    4. 3.4Comparing the outputs9 min
  4. 04

    The Six Solutions

    This book asks for one record per glass. A record says which pixels are that glass and where it stands on the table. What is asked for states the question in full. This chapter answers it in six different ways. No one of the six…

    7 sections · 30 min

    1. 4.1The six solutions — one question, six ways to see7 min
    2. 4.2Rules on the table4 min
    3. 4.3A network trained from scratch4 min
    4. 4.4A borrowed model, as it downloads4 min
    5. 4.5The same model, fine-tuned3 min
    6. 4.6A foundation model with a keeper4 min
    7. 4.7A transformer segmenter, fine-tuned4 min
  5. 05

    Solution 1: Rules on the Table

    This chapter is about the one answer to the problem this book sets that uses no model. It fits nothing. It states a rule that a person can read, and it applies that rule. It is for a reader who has read the shared setup pages and…

    6 sections · 44 min

    1. 5.1What it is5 min
    2. 5.2The code8 min
    3. 5.3How it works15 min
    4. 5.4A worked example9 min
    5. 5.5What it needs1 min
    6. 5.6How it compares6 min
  6. 06

    Solution 2: A Network Trained From Scratch

    This chapter is for a reader who has read the cell and the problem. The cell is described there, so these pages cover only this solution.

    6 sections · 43 min

    1. 6.1What it is4 min
    2. 6.2The code6 min
    3. 6.3How it works19 min
    4. 6.4A worked example4 min
    5. 6.5What it needs1 min
    6. 6.6How it compares9 min
  7. 07

    Solution 3: A Borrowed Model, as It Downloads

    This solution answers what this book asks for by downloading a model and running it. It collects no labels and trains nothing.

    3 sections · 21 min

    1. 7.1What it is4 min
    2. 7.2The code6 min
    3. 7.3Results and comparison11 min
  8. 08

    Solution 4: The Same Model, Fine-Tuned

    This chapter answers what this book asks for in a common way. It takes a model that was trained somewhere else, and trains it further on pictures of this cell. This is called fine-tuning. It is the usual way to use a borrowed…

    4 sections · 30 min

    1. 8.1What it is5 min
    2. 8.2The code5 min
    3. 8.3How the model is trained14 min
    4. 8.4Results and comparison6 min
  9. 09

    Solution 5: A Foundation Model With a Keeper

    This chapter is for a reader who has read the shared pages on the cell, the problem and the examiner. It describes solution 5. This solution does the job this book sets out with almost nothing trained in this cell.

    6 sections · 47 min

    1. 9.1What it is4 min
    2. 9.2The code5 min
    3. 9.3How it works21 min
    4. 9.4A worked example5 min
    5. 9.5What it needs2 min
    6. 9.6How it compares10 min
  10. 10

    Solution 6: A Transformer Segmenter, Fine-Tuned

    This is the sixth of the six solutions in this book. It uses the newest design of the six. It takes a transformer segmenter, RF-DETR-Seg, and trains it further on this cell's pictures.

    6 sections · 36 min

    1. 10.1What it is4 min
    2. 10.2The code7 min
    3. 10.3How it works11 min
    4. 10.4A worked example5 min
    5. 10.5What it needs2 min
    6. 10.6How it compares7 min
  11. 11

    The Results

    This page is the scoreboard for the six solutions in this book. It answers one question. With the same pictures and the same marking, how many glasses did each method find, and how good were its masks? It is for a reader who has…

    1 section · 10 min

    1. 11.1The six solutions side by side10 min
  12. 12

    How a Mask Becomes a Record

    Every solution in this book returns masks. A mask is the set of pixels that belong to one glass. But every scorecard reports places and widths. This page explains the step in between. It is the arithmetic the examiner uses to…

    1 section · 6 min

    1. 12.1How a mask becomes a record6 min
Next bookBook 9: Pushing the Glasses ApartOne manipulation problem, answered six ways and measured