Skip to content
Free public beta until December 4, 2026: every course is open, no card needed. After that, the first chapter of each course stays free and full access is a one-time $39 ($19 for up to 100 founding members chosen for helpful feedback). Your progress is kept. How the beta works

Recipe

Stop a run at the right time, and keep the right weights

Use when: you have a training loop already checking validation loss on some schedule (Section 4), and want to stop automatically once it stops improving, without losing the best version

Goal: the best-performing set of parameters this run reached, saved and ready to use, with training halted once continuing stopped helping

Taught in Early stopping and checkpoints

Quick recipe

  1. Pick a patience, P
  2. After every validation check, compare to the best so far
  3. On a new best, save a checkpoint and reset the counter
  4. Otherwise, increment the counter
  5. When training stops, use the saved checkpoint, not the final weights
  1. 1

    Choose how many consecutive non-improving validation checks to tolerate before stopping.

    Why: P is the one knob that trades "stop too early, miss a real improvement" against "keep training long past the real best, for nothing."

  2. 2

    If this check's validation loss beats every earlier one, it's a new best.

    Example

    Validation losses 1.30, 1.05, 0.92, 0.97, 0.90, 0.94: epoch 5's 0.90 is a new best, since it beats epoch 3's 0.92.

    Why: every later decision — resetting the counter, saving a checkpoint — depends on knowing whether THIS check is actually an improvement.

  3. 3

    Save \theta (and the rest of the checkpoint's contents) and set the non-improving counter back to 0.

    Why: this is the only copy of the network worth keeping if training later gets worse — losing it would mean losing the best result the whole run produced.

  4. 4

    Add 1 to the count of consecutive checks that failed to beat the best.

    • the counter reaches P — stop training now.
    • the counter is still below P — continue training normally.

    Why: this is what turns "stopped improving once" into "stopped improving for long enough that it's not coming back," without quitting after a single unlucky check.

  5. 5

    Load whichever checkpoint was saved at the best epoch, and discard the parameters training happened to end on.

    Why: the run keeps going for P more epochs after the true best, specifically to give a plateau a chance to end — so the weights at the moment of stopping are already worse than the saved best.

Common mistakes

  • Checking the patience counter against training loss instead of validation loss — training loss rarely stops improving, so the counter would almost never reach P at all.
  • Using the final epoch's weights after stopping instead of the saved best-so-far checkpoint.
  • Overwriting a single checkpoint file every epoch regardless of validation performance, leaving no way back to an earlier, better version once training gets worse.