Problem-solving recipes

A step-by-step procedure for every problem type this course teaches — what to do first, what to do next, and how to tell which case you're in.

Local features and keypoint detection

Classify a window with the Harris response

Use when: you have Sobel gradients $G_x,G_y$ over a small window and need to decide flat, edge, or corner without solving for eigenvalues

Goal: a single signed number $R$ that classifies the window

Corner detection: Harris, Shi-Tomasi, and FAST

Feature matching and RANSAC

Match features between two images

Use when: you have two images of a related scene and need point correspondences reliable enough to feed a downstream fit: a homography, a fundamental matrix, or any other relationship between the two views

Goal: a set of point correspondences clean enough that a downstream geometric fit can trust every one of them

The feature-matching recipe

Camera geometry and calibration

Calibrate a camera

Use when: you have a real camera whose intrinsics, extrinsics, and lens distortion are unknown. You also have a calibration target (often a checkerboard) whose true geometry you already know exactly

Goal: a camera matrix P = K[R|t] per observed pose, plus distortion coefficients. Both need to be good enough that a downstream computation (Chapters 6, 8, 9) can trust every pixel measurement this camera produces

The camera calibration recipe

Stereo vision and 3D reconstruction

Reconstruct a 3D scene from a stereo pair

Use when: you have two images of the same scene from calibrated cameras (known intrinsics and relative pose, Chapter 7). You want an actual 3D reconstruction, not just a 2D correspondence

Goal: a point cloud. That's a set of 3D scene points, each triangulated from a matched pair of rectified image points

Dense vs. sparse correspondence, and point clouds

Neurons, training, and CNNs for images

Count a layer's parameters and its output size

Use when: you're given a convolutional or pooling layer's kernel size, channel counts, and stride/padding, and asked how many parameters it has or what shape it outputs

Goal: the layer's total parameter count and the spatial size of its output

Convolution as a learned operation, pooling, and receptive fields

Transfer learning and fine-tuning a pretrained backbone

Use when: you have a backbone pretrained on a large source task and a new target task with limited labeled data

Goal: a working classifier for the new task, trained in far less time and data than training from scratch

Transfer learning and fine-tuning

Build an image classifier

Use when: you're handed a labeled image dataset and asked to build a working classifier from it

Goal: a trained, honestly-evaluated image classifier, with a documented reason behind every major choice along the way

Transfer learning and fine-tuning

Object detection

Run the object detection pipeline end to end

Use when: you have a trained detector and a new image, and need a final, clean list of detected objects

Goal: a set of final detections, each with a class, a box, and a confidence score, cleaned of duplicates and thresholded to a usable confidence level

Precision, recall, and mAP

Semantic and instance segmentation

Turn per-pixel class scores into a segmentation map

Use when: a network has already produced a raw score for every class, at every pixel, and you need the actual labeled map

Goal: a class map the same height and width as the input image, one class label per pixel

Semantic vs. instance segmentation

Produce an instance segmentation map with Mask R-CNN

Use when: you need a separate mask for each individual object, not just a per-class map

Goal: a list of detected objects, each with its own class, confidence score, and pixel mask

FCN, U-Net, and Mask R-CNN

Image embeddings and retrieval

Retrieve images similar to a query

Use when: you have a query image, a gallery of candidate images, and need the closest matches

Goal: a ranked list of the k closest gallery images, with their similarity scores

Retrieval pipelines and why generic embeddings win

Search a gallery of embeddings for nearest neighbors

Use when: you have a query embedding and a gallery of precomputed embeddings, and must decide HOW to search

Goal: the correct (or a known-accurate approximate) top-k neighbor set, using a strategy sized to the gallery

Retrieval pipelines and why generic embeddings win

Optical character recognition and document vision

Run the OCR pipeline end to end

Use when: You have a photographed or scanned document. You need its text as clean, structured output, not a pile of separate strings.

Goal: A structured result: a form's labeled fields, a table's rows and columns, or an ordered block of paragraphs. It's built from every region's recognized text, with low-confidence regions flagged rather than silently trusted.

The complete pipeline, worked end to end

Video, motion, and tracking

Track objects across frames

Use when: you already have per-frame object detections (Chapter 13) and need to give each one a consistent identity across a video, including through brief occlusion

Goal: every existing track either updated with a matched detection, still predicting through an occlusion, or cleanly dropped, plus a fresh track for anything genuinely new

Multi-object tracking and occlusion

Human pose and depth/3D vision

Run 2D/3D pose estimation on an image

Use when: you have an image (or short frame sequence) with one or more people and need each person's own skeleton, not just a bounding box

Goal: a full skeleton per person -- every keypoint located (or explicitly flagged as unreliable) and connected in the correct anatomical order

2D/3D pose keypoints and skeletons