The map of techniques

This book describes 34 programming techniques, and they have different names, take in different things and give back different answers. This page is the map of all of them, because it sorts them into seven kinds, which this book calls categories. For each category it then says what the techniques do and where to read about them.

It answers three questions. What are the seven categories, and which techniques are in each? Where does each category do its job when one robot arm does one task? And in what order should you read the chapters of this book?

It is for a reader who has read the earlier pages of this chapter, especially programmed, not learned. You can also come back to it at any time, when you want to see where one technique fits among the others. Book 6 has a page with the same shape for learned models, the map of models.

Contents#

1. One task, seven kinds of technique
2. The seven categories · Geometry and cameras · Searching and matching · Fitting and estimation · Image and point cloud processing · Planning and search · Control and motion · Decisions and task logic
3. All 34 techniques in one table
4. When each kind does its job
5. How the categories connect
6. Find a method by job
7. A suggested reading order
8. Where to read next
9. Using it in Python

1. One task, seven kinds of technique#

The easiest way to see all seven categories is to follow one task from start to finish. A robot arm has a camera on its wrist, and there are three mugs on a table with a rack of pegs at one end. The task is to find the mugs with the wrist camera and hang each one on the rack.

To do this, the robot must do several separate things. It must turn what the camera sees into positions measured from its own base, and it must pick the mug pixels out of the picture and ignore the table. It must find the table top and keep each mug's position steady while the camera moves, and then decide which mug is which from one picture to the next. After that it must find a path to each mug and on to the rack that hits nothing, and drive its motors smoothly along that path. Finally it must keep track of the whole job: which mug is next, and what to do if a grasp fails.

Each of these jobs is done by a different kind of technique, and those kinds are the seven categories.

The picture shows the task with the seven categories numbered; each number sits next to the part of the scene that its techniques work on: the camera, the mugs, the table top, the mug pixels, the path to the rack, the joints and the task list.

A real arm does not always use every technique in this book, and a simple pick-and-place arm may use only six or seven of them. However, almost every arm task uses at least one technique from each of the seven categories.

2. The seven categories#

Section 1 numbered the seven categories on the picture, so this section describes each of them in turn. Each category below has one paragraph that says what its techniques do, followed by a link to the chapter overview, which explains each technique in one line and compares them.

Inside each chapter, the technique pages are split into two groups. The most used group holds the techniques that nearly every arm program needs, or that matter most when they go wrong. However, the also used group holds techniques that are used often, but only for some tasks or some kinds of arm. So read the most used group of a chapter first, and each paragraph below names the most used techniques first and then the also used ones.

Geometry and cameras#

Geometry and cameras turn pixels, frames and joint angles into positions you can trust, and four of these techniques are most used. The pinhole camera model says which pixel a point in space lands on, and turns a pixel and its depth back into a point. Rigid transforms move a position from one frame to another, such as from the camera to the arm's base. Calibration measures the numbers that the other two need: the camera's own settings, and where the camera sits on the arm. Pose from points finds where a known object sits from a few of its points seen in one picture, a problem called perspective-n-point (PnP). One technique is also used: multi-view geometry combines pictures taken from different places to find where points are in space. So in the mug task, these techniques turn each mug's pixels into a position the arm can reach. Read the geometry and cameras overview.

Searching and matching#

Searching and matching find the closest thing, and decide which thing is which, and three of these techniques are most used. Nearest-neighbour search finds the point or object closest to a given one, as in the closest-mug example on the first page. Iterative closest point lines up two sets of points, such as a stored 3D model of a mug and a fresh scan. Assignment and matching pairs up two lists, such as the mugs seen in this picture and the mugs seen in the last one. One technique is also used: image features and matching finds the same small spots in two pictures and pairs them up. So in the mug task, these techniques make sure that "mug 2" in one picture is still "mug 2" in the next. Read the searching and matching overview.

Fitting and estimation#

Fitting and estimation get a clean shape or a steady number out of noisy measurements, and four of these techniques are most used. Least-squares fitting finds the line, plane or circle that is closest to many points. Random sample consensus (RANSAC) does the same while ignoring points that are plainly wrong. The Kalman filter combines each new reading with what it already knew, to give a steady estimate of something that may be moving. Sensor streams covers how to handle readings that arrive one after another from many sensors, at different rates and times. One technique is also used: system identification measures the numbers inside a physical model of the arm, such as a joint's friction or the mass in the gripper. So in the mug task, RANSAC finds the table top, and a Kalman filter keeps each mug's position steady as the camera moves. Read the fitting and estimation overview.

Image and point cloud processing#

Image and point cloud processing clean up and cut up pictures and point clouds so objects stand out, and three of these techniques are most used. Thresholding and colour masks keep the pixels whose colour or depth is in a chosen range. Morphology and the distance transform tidy up those pixels, and find the point most central in a shape. Clustering groups nearby points into separate objects. Two techniques are also used, and the first of them, edges and contours, finds the outline of each shape. The second, volumetric maps, divides the space around the arm into small cubes and records which ones are full, empty or not yet seen. So in the mug task, a depth threshold removes the table, and clustering splits the remaining points into one group per mug. Read the image and point cloud processing overview.

Planning and search#

Planning and search find a way for the arm to get from here to there without hitting anything, and three of these techniques are most used. Sampling-based planning tries random arm poses and joins the safe ones into a path. Numerical inverse kinematics finds the joint angles that put the gripper at a chosen pose. Trajectory optimisation takes a path and makes it shorter, smoother and further from obstacles. Three techniques are also used, and the first of them, graph search, finds the shortest route through a grid or a graph of poses. Sampling-based optimisation and model predictive control (MPC) tries many random plans, keeps the best, and plans again a moment later from where the arm now is. Visibility and next best view works out what a camera can see from a pose, and chooses where to look next. So in the mug task, inverse kinematics turns "gripper above mug 2" into joint angles, and a planner finds a path to the rack that does not hit the rack itself. Read the planning and search overview.

Control and motion#

Control and motion turn a planned path into smooth, safe motor commands, and four of these techniques are most used. Proportional-integral-derivative (PID) control makes each joint follow its target angle, many times a second. Trajectory generation decides how fast to move along the path at each moment, so the arm starts and stops smoothly. Arm dynamics works out the torque each joint needs to hold the arm up and to speed it up, so the controller does not have to wait for an error. Safety monitoring checks the arm's limits in software and stops it when one is broken or a person comes too close. One technique is also used: impedance and force control makes the arm give way when it touches something, and stop if it pushes too hard. So in the mug task, these techniques move the arm to each mug, and let it feel the peg when it hangs the mug on the rack. Read the control and motion overview.

Decisions and task logic#

Decisions and task logic decide what the robot does next, and in what order, and two of these techniques are most used. A finite state machine moves the robot from one named step to the next, such as "looking", "picking" and "hanging". A behaviour tree arranges the steps as a tree, which makes it easier to add retries and fallbacks. Two techniques are also used, and the first of them, greedy algorithms and set cover, makes a good choice quickly by taking the best-looking option at each step. Optimisation solvers find the best order or assignment when the greedy choice is not good enough. So in the mug task, a state machine runs the job, and a greedy rule picks the closest mug next. Read the decisions and task logic overview.

3. All 34 techniques in one table#

Section 2 described the categories one at a time, and the table below now gathers every technique page in the book into one place, in the order of the chapters. Read each row across: the category, the technique, whether it is in the chapter's most used or also used group, what it does in one line, and what it does in the mug task. The technique names are links to their pages.

So of the 34 pages, 23 are in a most used group and 11 are in an also used group.

A few of these do the same job in two ways. For example, finite state machines and behaviour trees both run the task, least squares and RANSAC both fit shapes, and graph search and sampling-based planning both find paths. So the choosing a technique page explains how to pick between them, and each technique page compares itself with its neighbour.

4. When each kind does its job#

The map picture in section 1 shows where each category works, but it is also useful to see when each one works. So the task for one mug can be split into seven steps: look from above, find the mugs, choose the next mug, plan the reach, reach and grasp, carry the mug to the rack, and hang it and let go.

Each coloured bar shows the steps during which one category is busy; decisions and task logic run for the whole task, perception techniques are busy at the start, and control is busy from the reach to the end.

Read the picture from left to right. At the start, image and point cloud processing and geometry turn the camera's pictures into mug positions. Fitting and searching then tidy those positions and match them to the mugs seen before, and planning works out the reach. After that, control drives the arm for the rest of the task. Geometry and fitting come back during the grasp, because the camera takes a closer look as the gripper comes down. Decisions and task logic run from start to finish, because they decide when every other step begins.

This is one way to build the task, not the only one. For example, a task where the mugs are always in the same place would not need the perception bars at all. However, a task where the arm must avoid people would have planning running during the carry as well.

5. How the categories connect#

Section 4 showed when each category is busy, and this section shows how they hand work to each other. The seven categories are separate chapters, but the techniques in them depend on each other, because the output of one is very often the input of the next.

First a threshold and clustering cut the mugs out of the depth picture, and then the pinhole camera model turns each mug's pixels into points. A rigid transform moves those points into the base frame, and RANSAC and least squares fit the table and the mug's rim. Nearest-neighbour search and assignment then keep track of which mug is which, so that a greedy rule can choose the next mug. After that, inverse kinematics and a planner find the joint angles and the path. Then trajectory generation and PID control drive the motors, with arm dynamics supplying most of the torque and safety monitoring watching every command. A state machine or behaviour tree runs all of it in order, and this chain, from picture to motors, is the most common way to build a programmed picking robot.

Many of the categories also share the building blocks from the building blocks. Planning and decisions both use graphs, while fitting, planning and optimisation solvers all minimise a cost function, and geometry and planning both use transforms. So once you know one chapter, the next one is easier.

The categories also connect to Book 6, because some steps in the chain are often done by a learned model instead of a written technique. The table below shows the most common swaps, so read each row as one job: the written technique in this book, and the kind of learned model that can replace it.

Even when a learned model takes over one job, the written techniques around it usually stay. For example, a detection model gives only a box in the picture, so the pinhole camera model and a rigid transform still have to turn that box into a position the arm can reach.

6. Find a method by job#

Most people come to these books with a job in mind rather than a method, so this section starts from the job instead of the technique. Read each row across: a job the arm must do, the Book 2 or Book 3 page that helps you choose how to do it, the written techniques in this book that can do it, and the learned models in Learned Models that can do it.

Most jobs have both a written and a learned answer, and the Book 2 or Book 3 page says which one suits which case. However, two rows have only one side. No learned model is in common use to check whether the arm can reach a pose, because inverse kinematics already gives an exact answer quickly. And no written technique can follow an instruction in free wording, because a person can say the same thing in too many ways to list them all. Many real arms therefore mix the two sides, so that a learned model finds the object and written techniques do the rest.

The learned column also includes the methods of Book 7's classical machine learning chapter, which are not neural networks. They appear in the rows where the input is only a few measured numbers: correcting a sensor, tuning a controller's gains, learning a motion from a few demonstrations, and scoring grasps or spotting faults from logged numbers.

7. A suggested reading order#

Section 6 lets you jump to a technique by job, but if you want to read the book through, the chapters are numbered in the order this book suggests. Read them from 01 to 08, which is the same order as the folders. Each chapter explains its own terms, so you can also start anywhere, and the list below gives the reason for each step.

1. This chapter, starting with programmed, not learned. Everything else uses its words: technique, parameter, frame, graph, noise, cost and loop.
2. Geometry and cameras. Every other chapter needs positions in the right frame.
3. Searching and matching. It finds the closest thing and decides which thing is which.
4. Fitting and estimation. It turns noisy pixels and points into clean shapes and steady numbers.
5. Image and point cloud processing. It cuts the objects out of a picture or a point cloud.
6. Planning and search. It needs positions and shapes from the chapters before it.
7. Control and motion. It carries out what the planner decided.
8. Decisions and task logic. It ties all the others together into a whole task.

Chapters 03, 04 and 05 lean on each other, so you can read those three in any order, because each one uses a little of the other two. For example, clustering in chapter 05 uses the nearest-neighbour search from chapter 03, while RANSAC in chapter 04 is often run on the points that chapter 05 cut out. And the matching in chapter 03 often works on positions that a Kalman filter from chapter 04 has made steady. So whichever of the three you read first, the other two will be easier after it.

So if you have one job in mind, you can jump straight to its chapter. For example, if you only want the arm to move smoothly to a known pose, read planning and search, and then control and motion.

8. Where to read next#

• The geometry and cameras overview is the next chapter in the suggested order.
• The map of models in Book 6 does the same job for learned models.
• Programmed methods for one arm in Book 3 shows how these techniques are combined into whole systems.
• Tools and libraries in Book 3 shows the software, such as MoveIt 2, OpenCV and Open3D, that provides many of these techniques ready to call.

9. Using it in Python#

Section 3 listed all 34 techniques in one table and section 6 lets you find one by the job you have. This section adds the practical layer under both of them, which is the library you would actually import for each of the seven categories. After reading it you should be able to look at a job on your arm and know which package to install first.

Each line below is the usual starting point for one category in Python, and the whole block is a real set of imports rather than a picture of one.

import cv2                                       # geometry and cameras: cv2.solvePnP
from scipy.spatial import KDTree                 # searching and matching: tree.query
from scipy.optimize import least_squares         # fitting and estimation: a robust fit
import open3d as o3d                             # image and point cloud processing
from moveit.planning import MoveItPy             # planning and search, through ROS 2
from ruckig import Ruckig, InputParameter, OutputParameter   # control and motion
import py_trees                                  # decisions and task logic

Those seven cover the mug task from section 1 almost completely. OpenCV and Open3D turn the wrist camera's pictures into mug positions, SciPy's KDTree and least_squares tidy those positions and match each mug to the mug seen in the last picture, MoveItPy plans a reach that hits nothing, Ruckig turns the plan into joint commands that respect the arm's speed and acceleration limits, and py_trees holds the order of the whole job. Book 3's tools and libraries shows working code for MoveItPy and py_trees on a six-joint arm.

What the libraries do for you is the hard arithmetic inside each technique, which is why this book explains the steps rather than asking you to write them. What you write yourself is everything between the seven lines: the code that carries numbers from one library to the next in the right frame and the right unit, and the code that decides what to do when a step returns nothing. That glue is usually larger than all seven calls together, and it is where the bugs live, because no library checks that the pose you hand to MoveItPy is measured from the base rather than from the camera.

What you have to decide first is which of the seven you need at all. As section 1 said, a simple pick-and-place arm uses only six or seven of the 34 techniques, so importing all seven libraries into one program before you need them buys you trouble. It is real trouble, too, because these packages do not arrive together: MoveItPy and Ruckig come with a ROS 2 installation, while OpenCV and Open3D come from pip or conda, and two of them in one Python process can insist on different versions of NumPy. Adding one library at a time, and checking the arm still runs after each one, is slower to start and much faster to finish.