Recipe

Reconstruct a 3D scene from a stereo pair

Use when: you have two images of the same scene from calibrated cameras (known intrinsics and relative pose, Chapter 7). You want an actual 3D reconstruction, not just a 2D correspondence

Goal: a point cloud. That's a set of 3D scene points, each triangulated from a matched pair of rectified image points

Taught in Dense vs. sparse correspondence, and point clouds

1. 1

   Use each camera's known intrinsics and the pair's relative pose R, \mathbf{t} (Chapter 7). Compute the rectifying warps. They make every pair of corresponding epipolar lines land on the same horizontal row in both images.

   Why: Rectification turns Section 1's 2D epipolar-line search into a 1D search along one shared row. That's what makes "disparity" a single horizontal number instead of a full 2D offset.

2. 2

   Run Chapter 5's own detect \to describe \to match \to ratio-test \to RANSAC pipeline on the rectified pair. Search only along each keypoint's own shared row, instead of the whole other image.

   • the goal is a sparse point cloud (e.g., feeding structure from motion, Chapter 9) — keypoint correspondences alone are enough.
   • the goal is a dense depth map — replace keypoint matching with a dense stereo-matching algorithm. It searches for a disparity at every pixel, not only at keypoints (this section, above).

   Why: Rectification only pays off once the search actually uses it. Searching the whole image after rectifying wastes the reduction rectification bought.

3. 3

   For each correspondence, take the difference in horizontal pixel position between the two rectified images: d = x_1 - x_2.

   Example

   Matched at x_1=3 in image 1 and x_2=1 in image 2 (Section 2's own worked example): d = 3 - 1 = 2.

   Why: Disparity is the one number a rectified pair actually measures directly. Every remaining step is just math on d, not a new search.

4. 4

   Convert each disparity into depth with Z = fb/d, then recover X and Y from the matched image coordinates and Z.

   Example

   With b=8, f=4, d=2: Z=(4)(8)/2=16. With x_1=3, y=1.5: X=x_1 Z/f=12, Y=yZ/f=6 -- the same (12, 6, 16) Section 2 found.

   Why: This is the geometric inverse of projection. It turns a 2D correspondence pair, plus the known camera setup, back into one real 3D point.

5. 5

   Repeat Steps 2-4 for every correspondence found (sparse) or every pixel (dense) to build the complete set of reconstructed 3D points.

   Why: A single triangulated point is just a coordinate. The point cloud, assembled from every correspondence, is what actually represents the scene's geometry.