Skip to content
Free public beta until December 4, 2026: every course is open, no card needed. After that, the first chapter of each course stays free and full access is a one-time $39 ($19 for up to 100 founding members chosen for helpful feedback). Your progress is kept. How the beta works

Recipe

Track objects across frames

Use when: you already have per-frame object detections (Chapter 13) and need to give each one a consistent identity across a video, including through brief occlusion

Goal: every existing track either updated with a matched detection, still predicting through an occlusion, or cleanly dropped, plus a fresh track for anything genuinely new

Taught in Multi-object tracking and occlusion

Quick recipe

  1. Detect objects in the current frame
  2. Predict each track's next position
  3. Associate new detections with existing tracks
  4. Handle unmatched detections and unmatched tracks
  5. Update every matched track's state
  6. Drop a track that's been unmatched too long
  1. 1

    Run Chapter 13's detector on this frame. Collect its list of scored boxes.

    Why: tracking has nothing to associate or correct without a fresh set of per-frame detections. It consumes them as input; it doesn't produce them.

  2. 2

    For every track already being followed, use its own Kalman filter's motion model (section 2) to project its state forward one frame. Do this independently for every track.

    Why: a prediction is what makes matching possible at all. Without it, there's nothing to compare this frame's detections against.

  3. 3

    Compute the IoU between every predicted track box and every new detection box. Match the highest-scoring pairs first, never reusing a track or detection that's already matched. (Or solve it optimally with the Hungarian algorithm instead.)

    Example

    Track 7 predicted at IoU 0.87 against detection-a; track 12 predicted at IoU 0.72 against detection-b. Both clear a 0.5 threshold and get matched. Detection-c matches neither (IoU 0 against both).

    Why: IoU is the same overlap measure Chapter 13 already used to score a box against ground truth. Here, it scores a predicted box against a fresh detection instead.

  4. 4

    Not every detection finds a track, and not every track finds a detection. Both leftovers need an explicit rule, rather than being silently dropped or ignored.

    • a new detection has no matching track — start a new track from it.
    • a track has no matching detection — mark it occluded. Keep predicting from its motion model, with no correction, rather than deleting it right away.

    Why: a real scene constantly has new objects entering, and objects briefly hidden behind something else. Treating either case as an error would either lose every genuinely new object, or destroy a track's identity the instant it's occluded for even one frame.

  5. 5

    For each track matched to a detection this frame, run section 2's correct step. Blend the predicted state with the new measurement, weighted by the Kalman gain.

    Why: this is the same predict-then-correct cycle section 2 already taught. It's now running once per matched track, instead of once for a single object.

  6. 6

    Suppose an occluded track has gone unmatched for more than a small, fixed number of frames -- a few seconds' worth. Then remove it, instead of continuing to predict from it forever.

    Why: predicting from an old motion model with no new evidence gets less reliable the longer it runs unconfirmed. Eventually, the object has probably left the scene, or its motion has changed enough that the old prediction can no longer be trusted.

Common mistakes

  • Deleting a track the instant one frame goes unmatched, instead of allowing a short occluded window. This breaks identity on every ordinary brief occlusion, which is common, not rare.
  • Never dropping an occluded track at all. An old, stale prediction eventually drifts arbitrarily far from where the real object actually is.
  • Matching by proximity alone, with no IoU/overlap check. This breaks down the moment two tracked objects pass close to each other.
  • Forgetting that matching has to run before either the correct step or the new-track/drop decisions. Those later steps all depend on knowing which detection, if any, belongs to which track first.