Skip to content
Free public beta until December 4, 2026: every course is open, no card needed. After that, the first chapter of each course stays free and full access is a one-time $39 ($19 for up to 100 founding members chosen for helpful feedback). Your progress is kept. How the beta works

Recipe

Diagnose a failing model

Use when: a trained model underperforms and you need to find the actual root cause, not just retune blindly

Goal: the correct root cause, found in the order it is most likely to be found, without wasted effort chasing the wrong fix

Taught in Why models fail: a troubleshooting order

Quick recipe

  1. Check for data leakage
  2. Check the train/test split itself
  3. Check underfitting vs. overfitting
  4. Check the metric
  5. Check that runs are stable
  1. 1

    Look for any feature, preprocessing step, or split that let target information leak into training.

    Example

    An oddly high validation score that collapses in production is the classic sign.

    Why: Leakage is the most common and most harmful cause. It can make every later problem look like something else entirely.

  2. 2

    Confirm the split respects grouping, time order, or independence in the data. Don't assume a plain random split was safe.

    Why: A bad split can silently let related or future samples leak between training and testing. This can look like good performance that won't hold up.

  3. 3

    Plot a learning curve (Chapter 11) instead. Don't just trust a single accuracy number.

    Why: A single score cannot tell apart a model that is too simple from one that has memorized the training set.

  4. 4

    Confirm the metric fits the problem. Accuracy on an imbalanced dataset (Chapter 13) can look good while being meaningless.

    Why: A correctly working model can still be judged by a metric that hides the real failure.

  5. 5

    Confirm the random seed is fixed and results are stable across repeated runs.

    Why: Without this check, "the model got worse" can't be told apart from ordinary run-to-run variance.

Common mistakes

  • Retuning hyperparameters before ruling out leakage or a bad split wastes effort. It risks a worse production failure.
  • Trusting a single accuracy number instead of a learning curve to judge under/overfitting.
  • Judging results by a metric that doesn't fit the problem is a mistake. One example: plain accuracy under severe class imbalance.