Multi-view geometry: depth from two pictures

This page explains how a program finds depth from two pictures of the same scene, without a depth sensor. One picture only tells you the direction of each point, but two pictures, taken from two known places, tell you where the point is. The page covers five ideas, and the first two of them work together. Triangulation finds a point from two lines of sight, and epipolar lines say where to look for a match in the second picture. Rectified stereo turns the shift between two side-by-side pictures into depth, while parallax finds a height from how far a point shifts when the camera slides. And the ground-plane constraint gets depth from a single camera when the point is known to be on the table.

It is for a reader who has read the pinhole camera model and rigid transforms. Book 2 already uses some of these ideas. Baseline, not count shows why two views far apart beat many views close together, and two photos from one moving camera measures an object's width and height by moving the camera. So this page explains the geometry underneath both. Every number here comes from a real run of the diagram script, docs/diagrams/geometry_and_cameras_2.py, with Book 2's camera: 320 × 240 pixels, fx = fy = 277.1, cx = 160, cy = 120, 0.40 m above the table.

Contents#

1. The idea in one sentence
2. How it works, step by step · Triangulation: where two rays cross · Epipolar lines: where to look for the match · The essential matrix, in plain words · Rectified stereo: disparity to depth · Parallax: height from a sliding camera · The ground-plane constraint: one camera and a known table · Pseudocode
3. Where it is used on a robot arm
4. Where it works, and where it does not
5. Libraries that provide it
6. Why two views, and what it costs
7. The learned alternative
8. Where to read next
9. Using it in Python

1. The idea in one sentence#

A pixel tells you the direction of a point; two pixels of the same point, seen from two known places, tell you where the two directions cross, and that crossing is the point.

So here is an everyday example: hold a finger up in front of your face, then close your left eye and then your right. The finger jumps sideways against the wall behind it, and a near finger jumps a lot while a far one jumps a little. Your brain uses that jump to judge distance, and two cameras, or one camera that moves, can do the same, because they can measure the jump in pixels.

Everything on this page depends on knowing where the two viewpoints are relative to each other. On a robot arm that knowledge is often free, because the arm itself moved the camera, and its joints report exactly how far.

2. How it works, step by step#

Triangulation: where two rays cross#

Section 1 said that two directions cross at the point, so this section starts with the arithmetic of that crossing. Each pixel defines a ray, which is a straight line that starts at the camera and runs out into the scene. The pinhole camera model gives its direction, ((u - cx) / fx, (v - cy) / fy, 1) in the camera's frame, and the camera's pose turns that direction into the room's frame.

Triangulation means finding the point where two rays cross. With perfect numbers they cross exactly, but with real, noisy pixels two rays in 3D usually miss each other by a little. So the program looks for the point that is closest to both rays. The standard way writes two straight-line equations for each view and solves them together, the same kind of step as the direct linear first guess in pose from points.

Here is a worked example, in which a wrist camera looks straight down from 0.40 m and sees Book 2's spot on top of the red box at pixel (212.5, 86.5). The arm then moves the camera 80 mm to the side and turns it 13.2° so that it still faces the box, and in the second picture the spot is at pixel (211.9, 86.0).

On the left, the two rays cross on top of the box, and triangulating the two exact pixels gives (64.4, 41.1, 60.0) mm, which is the true spot. On the right is a close-up, where we moved each pixel by 0.5 pixels, in opposite directions. The rays now cross at (64.0, 40.5, 65.1) mm, which is 5 mm higher. Almost all of the error is along the line of sight, because two rays that meet at a narrow angle slide along each other easily. The dashed lines show each ray moved by 1.5 pixels, and the diamond between them is the region where the answer could land.

The distance between the two viewpoints is the baseline, and a wider baseline makes the rays meet at a wider angle, which makes the diamond shorter. So we repeated the example 2,000 times with 0.5 pixels of random noise on each pixel, for four baselines. Read each row as one baseline and how far the triangulated point landed from the truth.

The run agrees with the formula from Book 2's baseline, not count, with z = 0.34 m and e = 0.5 pixels, so doubling the baseline halves the error.

Epipolar lines: where to look for the match#

Triangulation needs to know which pixel in picture 2 shows the same point as a pixel in picture 1. Finding that pixel is called matching, and it is the hard part, but geometry makes it much easier.

The pixel in picture 1 is a ray, and the point is somewhere along that ray, although we do not know where. So now look at that whole ray from camera 2. A straight line in the room looks like a straight line in a picture, so the match must lie somewhere on one line in picture 2. That line is called the epipolar line, and it shrinks the search for the match from the whole picture to one line.

For a second worked example, camera 2 is now 80 mm to the side, 30 mm forward, and turned 13.2°. There is also a second red box on the table that looks the same as the first one.

The spot is at (212.5, 86.5) in view 1, and its epipolar line in view 2 is the purple line. The true match, at (211.9, 110.8), lies on that line, 0.00 pixels away, while the look-alike box is 56.0 pixels from it and therefore cannot be the match, however similar it looks. Book 2 uses exactly this test to match objects between two views in matching two views taken at one moment.

However, the test fails when two candidates both lie on or near the line. Then geometry alone cannot choose, and the program needs appearance or a third view.

The essential matrix, in plain words#

The epipolar line comes from one 3 × 3 matrix called the essential matrix, E, which holds how camera 2 is turned and shifted relative to camera 1. Write a pixel with the lens numbers taken out, x = ((u - cx) / fx, (v - cy) / fy, 1). Then for any point seen in both pictures,

x2ᵀ · E · x1 = 0,    where E = [t]× · R

R is the turn from camera 1 to camera 2, and t is the shift. [t]× is a small matrix that does a cross product with t. The equation says one thing in plain words: the two rays and the line between the two cameras all lie in one flat plane. They must, because they form a triangle with the point at its tip. So E · x1 is the epipolar line in picture 2, in lens-free units.

In our example, t = (-77.9, 30.0, 18.3) mm in camera 2's frame, and the equation gives -3.5 × 10⁻¹⁸ for the true match, which is zero apart from rounding. When the pixels still carry the lens numbers, the same matrix is written with them folded in: F = K⁻ᵀ · E · K⁻¹. F is called the fundamental matrix, and K is the 3 × 3 matrix of lens numbers.

The essential matrix can also run backwards. Given five or more matched points, and no knowledge of how the camera moved, a program can find E, and from it R and the direction of t. This is how a camera works out its own movement from pictures alone, and it always uses RANSAC, because some matches are wrong. However, it cannot find the length of t, because two pictures of a small scene taken close together look exactly like two pictures of a large scene taken far apart. So the scale must come from somewhere else: the arm's own movement, a known object size, or a depth reading. On a robot arm, the arm's joints almost always know the movement, so the program uses the known R and t directly.

Rectified stereo: disparity to depth#

A stereo camera is two cameras side by side, fixed to one bar, and the distance between them is its baseline. When the two cameras point the same way and their rows line up, the pair is called rectified. Real stereo cameras are never built perfectly, so their pictures are corrected in software to make them rectified, and that correction uses a stereo calibration.

In a rectified pair, every epipolar line is simply a row of pixels, so a point in row 86 of the left picture is in row 86 of the right picture. The only difference is how far along the row it appears, and that difference is the disparity, which is u_left minus u_right, measured in pixels. Depth then follows from one division:

depth = fx · b / d

So here is the worked example, with a 50 mm baseline, which is typical of a small stereo depth camera.

The spot on the box top is at u = 212.5 in the left picture and 171.7 in the right one, in the same row. So the disparity is 40.75 pixels, and 277.1 × 0.050 / 40.75 = 0.340 m, which is the right depth.

However, the chart on the right shows the catch. Depth falls as one over the disparity, so a far point has a small disparity, and a one-pixel mistake there costs a lot. The table gives the numbers, so read each row as one depth: its disparity, and the depth change caused by a mistake of one pixel and of a quarter of a pixel. Good stereo matchers reach about a quarter of a pixel.

The error grows with the square of the depth, so stereo is precise close up and poor far away. For a table-top arm, where most things are under 0.6 m away, it works well.

To get depth for every pixel, a stereo matcher searches each row of the right picture for the best match of each left pixel. That search needs texture, because a blank white table has nothing to match, and neither does a shiny surface that reflects different things into each camera. So many stereo depth cameras add a projector that throws a dot pattern onto the scene to give plain surfaces some texture. Book 2 compares these sensors in sensors.

Parallax: height from a sliding camera#

A wrist camera can make its own stereo pair. The arm slides the camera sideways, and the program compares the two pictures. How far a point moves in the picture is its parallax, which is the same thing as disparity, except that the baseline comes from the arm rather than from a fixed bar.

This is especially useful for heights. A camera looking straight down at a table sees every table point shift by the same amount, fx · b / H, where H is the camera's height. A point above the table is closer to the camera, so it shifts more, and that extra shift tells you how far above the table it is.

So here is the worked example. The camera is 0.40 m above the table and slides 40 mm. A mark on the table shifts 277.1 × 0.040 / 0.40 = 27.71 pixels. The spot on the box top shifts 32.60 pixels. So its depth is 277.1 × 0.040 / 32.60 = 0.340 m, and its height is 0.40 - 0.340 = 60.0 mm, the true height of the 6 cm cube.

A mistake of half a pixel in the shift gives 54.7 mm or 65.1 mm instead of 60.0, which is a 5 mm error. This is about the same size as in the triangulation example above, because it is the same geometry. A longer slide makes it smaller.

The table's shift is a useful check, because every table point should shift by 27.71 pixels. If they do not, the slide was not what the arm reported, or the camera was not looking straight down. Book 2 works through the same idea for an object's width in two photos from one moving camera.

The ground-plane constraint: one camera and a known table#

Sometimes one picture is enough. If a point is known to lie on the table, its depth is wherever its ray meets the table. So the table acts as the "second view", because it is a second surface that the point must lie on, and a ray and a plane cross at exactly one point. The pinhole camera model works the method through step by step.

However, the constraint is only as good as the claim that the point is on the table. Here is what happens when the claim is wrong. The straight-down camera sees the spot on top of the 6 cm cube at pixel (212.5, 86.5). If the program assumes the spot is on the table, it places it at (75.8, 48.4, 0) mm, while the truth is (64.4, 41.1, 60) mm. So the position on the table is 13.5 mm wrong, as well as the height. With the tilted camera from the pinhole page, which looks 60° down, the same 60 mm mistake puts a point 44.1 mm too far away. This is because a slanted ray travels a long way sideways for each millimetre of height.

So the ground-plane constraint suits flat things that really lie on the table: sheets, coins, the bottom edge of an object where it touches the table. For the top of an object, use one of the two-view methods above, or subtract the object's known height before intersecting the ray with a plane at that height.

Pseudocode#

So here are the three most common jobs of this page, written as plain steps.

# 1. Triangulate one point from two views with known poses
ray1 = T_base_camera1 * direction(pixel1, K)
ray2 = T_base_camera2 * direction(pixel2, K)
point = the point closest to both rays            # linear least squares (DLT)
gap = the shortest distance between the two rays
if gap > a few millimetres: reject the match       # the pixels were not the same point

# 2. Match along the epipolar line
E = skew(t) * R                                  # from the two known camera poses
F = inverse(K)ᵀ * E * inverse(K)
line = F * (u1, v1, 1)
candidates = features in picture 2 within 2 pixels of line
match = the candidate that looks most like the feature in picture 1

# 3. Height from a sideways slide of a downward camera
shift = u_after - u_before                        # in pixels, for the same point
depth = fx * slide_length / shift
height = camera_height - depth
check: table points must shift by fx * slide_length / camera_height

3. Where it is used on a robot arm#

Section 2 described five methods, and this section says where an arm uses them. Two views appear on a robot arm more often than it seems, and here are the concrete places.

• Stereo depth cameras. Most depth cameras on arms are stereo pairs with a dot projector. The rectification and the disparity search run inside the camera, and the depth picture that comes out is fx · b / d for every pixel.
• Measuring with a wrist camera. The arm moves the camera between two poses and triangulates corners, holes or edges. The baseline comes from the joints, so it is known exactly. Book 2's wrist camera page measures with this.
• Heights of objects a depth camera cannot see. Glass and shiny metal give no depth reading, but their outlines still shift with parallax.
• Matching the same object in two cameras. A cell with two fixed cameras uses epipolar lines to decide which detection in one picture is the same object as a detection in the other.
• Reading items on a flat table. A single colour camera above a conveyor or a table uses the ground-plane constraint to place flat parts, with no depth sensor.
• Building a 3D model of the scene. A camera that moves through many poses and triangulates many points builds a 3D model. Book 7's scene reconstruction covers the learned methods that do this.

4. Where it works, and where it does not#

The uses in section 3 all depend on knowing the two camera poses and matching the right pixels. Two-view geometry is exact, so its failures come from the inputs, and the table below lists the common ones. Read each row as what goes wrong, what you see, and what to do.

Two views also cannot see what neither camera sees, so a point hidden behind something in one view cannot be triangulated. And depth from stereo is poor far away, because the error grows with the square of the distance.

5. Libraries that provide it#

The table below lists well-known tools, so read each row as where to find it, which languages it covers, and what to call.

6. Why two views, and what it costs#

This section answers the four questions for two-view geometry: what it is, what it does for you, why it rather than the obvious alternative, and what it costs.

It is the geometry of two rays crossing. So it gives you depth and 3D points from ordinary colour pictures, and it tells you where to look for a match.

The obvious alternative is a single depth camera and nothing else, which gives a depth for every pixel with no matching step in your program. That is the right choice for most table-top arms. However, depth cameras fail on glass, shiny metal and black surfaces, and their fixed internal baseline sets their accuracy. Two-view geometry with a wrist camera instead works on any surface whose outline you can see, and lets you choose the baseline. For example, a 200 mm slide is about eleven times more precise than an 18 mm built-in stereo baseline. The ground-plane constraint needs no second view at all, but it only works for things that lie flat.

The cost is this. You need to know the two camera poses accurately, which means a good calibration and pictures taken while the arm is still. You need a reliable match for every point. You need a second picture, which takes time when the arm has to move. And you need a wide enough baseline, which the working space may not allow.

7. The learned alternative#

Book 7's depth from pictures covers two learned models for this job. A learned stereo model, such as RAFT-Stereo or FoundationStereo, does the matching with a network, then turns each shift into a depth with the same rule as this page, so its answer is still in real metres. Book 6 suggests it when you pick your own cameras, or work in bright light or at longer range, and like a written matcher it still struggles on a plain white surface. A monocular depth model, one that uses a single camera, guesses depth from one picture with no second view and no matching. But its errors are often several centimetres at one metre, many versions give no fixed scale, and Book 6 keeps it for rough jobs such as telling the foreground from the background. The geometry on this page still wins when you need a measured depth for a grasp, want no graphics processor, or can choose a wide baseline by moving the wrist camera.

8. Where to read next#

• The pinhole camera model gives the ray for each pixel, and the ray-and-plane method in full.
• Pose from points finds an object's pose when its shape is known, from one picture.
• Image features and matching finds the matched points that triangulation needs.
• RANSAC removes wrong matches before the essential matrix is fitted.
• Book 2's the wrist camera, end to end uses the baseline arithmetic to plan where to take the second picture.
• The overview shows where this page sits in the chapter.

9. Using it in Python#

Section 2 worked through triangulation, epipolar lines and rectified stereo with real numbers, and section 5 named the OpenCV functions that do each of them. This section shows the two calls you are most likely to write, because they cover the two different situations the page described. Triangulating a handful of matched points suits an arm that moves its own camera, while a dense disparity map suits a fixed stereo pair. After reading it you should know which of the two your robot needs.

Triangulation comes first. Each camera is described by a 3 by 4 projection matrix, which is K multiplied by the rotation and shift that carry a point from the world into that camera.

import cv2
import numpy as np

K = np.array([[277.1,   0.0, 160.0],
              [  0.0, 277.1, 120.0],
              [  0.0,   0.0,   1.0]])

def projection_matrix(R_world_to_cam, t_world_to_cam):
    return K @ np.hstack([R_world_to_cam, t_world_to_cam.reshape(3, 1)])

P1 = projection_matrix(np.eye(3), np.zeros(3))     # the first view defines the frame
P2 = projection_matrix(R2, t2)                     # the second, from the arm's joints

# One column per point, two rows: the pixel in each view.
pixels1 = np.array([[212.5], [86.5]])
pixels2 = np.array([[211.9], [86.0]])

homogeneous = cv2.triangulatePoints(P1, P2, pixels1, pixels2)
points_3d = (homogeneous[:3] / homogeneous[3]).T   # divide by the fourth number
print(points_3d)

A fixed stereo pair is the other case, and there the work is matching every pixel rather than a few. OpenCV's semi-global block matcher does that, and reprojectImageTo3D turns the result into points using the matrix Q that stereoRectify produced during calibration.

matcher = cv2.StereoSGBM_create(
    minDisparity=0,
    numDisparities=64,        # must be a multiple of 16; sets the nearest distance seen
    blockSize=5,
    P1=8 * 3 * 5 ** 2,        # the two smoothness penalties, from the OpenCV example
    P2=32 * 3 * 5 ** 2,
    uniquenessRatio=10,
    speckleWindowSize=100,
    speckleRange=2)

disparity = matcher.compute(left_rectified, right_rectified).astype(np.float32) / 16.0
points = cv2.reprojectImageTo3D(disparity, Q)

What the libraries do for you is the arithmetic that section 2 set out. triangulatePoints solves the two straight-line equations per view and returns the point closest to both rays, in the frame that P1 is written in, as four numbers that you divide through. StereoSGBM searches along the epipolar line for every pixel, which is possible only because rectification has already made those lines horizontal, and it returns the disparity in sixteenths of a pixel, which is why the code divides by 16.

What you still write yourself is what goes in. triangulatePoints needs the two pixels to be the same physical point, and finding that pairing is the job of image features and matching. It also needs both camera poses, which on an arm you read from tf2 at the moment each picture was taken. For the stereo path you have to run cv2.stereoCalibrate and cv2.stereoRectify first, keep the Q matrix they produce, and then mask out the pixels where the disparity came back as the "no match" value rather than letting them become points at an absurd distance.

What you have to decide or measure is the baseline and the search range. The baseline is the distance between the two viewpoints, and the table in section 2 shows what it buys: at 0.34 m, going from 18 mm to 80 mm took the median error from 11.3 mm to 2.5 mm, so this is the single most important choice on the page. numDisparities then sets how close an object can be and still be found, because a nearer object shifts further, and it costs time in proportion. blockSize trades noise against detail, since a larger block gives a smoother disparity map and loses the edges of small objects. All of these are measurements you make once on your own rig rather than numbers to copy.