Recipe

Debug a network that isn't learning

Use when: training loss barely moves, explodes, or a network's predictions look no better than guessing, and you don't yet know which of the usual causes is responsible

Goal: the specific, confirmed cause of the training failure, found by the cheapest and most isolating checks first

Taught in A fixed diagnostic order

1. 1

   Take 5–20 training examples. Turn off any regularization (dropout, weight decay). Train on just that tiny batch for a few hundred steps. A correct pipeline can always drive the loss on that tiny batch to nearly zero.

   Example

   A 4-layer classifier trained on 10 examples for 300 steps, dropout and weight decay both off. Loss starts at 2.30\approx\ln(10) (random guessing over 10 classes) and ends at 2.29. It never moved. Ten examples should be memorizable by almost any reasonable network in a few hundred steps. So this is a pipeline bug, not a tuning problem.

   • loss on the tiny batch drops to nearly zero — the pipeline itself is sound; continue to Step 2.
   • loss on the tiny batch stays flat or barely moves — the bug is in the pipeline itself, not in tuning. Go straight to Steps 3–7. Run each check against this same tiny batch, since whatever is broken there is broken everywhere.

   ⚡ Shortcut

   This single check is the fastest way through the whole recipe. Most training bugs never make it past Step 1. A broken loss function, a frozen layer, or mismatched labels all show up here in minutes. That's long before a learning-rate sweep or a gradient-norm log would have caught them.

   Why: a genuinely correct pipeline can always memorize a handful of examples. That pipeline is the forward pass, the loss, the backward pass, and the optimizer step together. Failing to memorize a handful of examples rules out "just needs more tuning." It points straight at the mechanics instead.

2. 2

   On the real dataset, try the current learning rate against one 10× higher and one 10× lower. Run each for a handful of epochs. Watch how much the loss curve's shape changes.

   Example

   Current \eta=0.001: loss goes from 2.30 to 2.27 over 20 epochs. At \eta=0.01: loss goes from 2.30 to 1.85 over the same 20 epochs. That's a real difference. It means the learning rate genuinely matters here. So this is where the fix belongs.

   • the loss curve changes noticeably with a 10× change in either direction — the learning rate was the bottleneck; tune it properly (see Chapter 11's training loop).
   • the loss curve barely responds to a 10× change either way — the learning rate isn't the cause; continue to Step 3.

   Why: a run that's insensitive to a full order-of-magnitude change in step size isn't being limited by step size at all. Something else is capping how much the network can learn, regardless of \eta.

3. 3

   Look at the very first forward pass, before any training step happens. Are the first layer's activations saturated at their extreme values? Is the first layer's gradient nonzero?

   Example

   A network whose weights were all initialized to exactly 0: every neuron in a layer computes the identical output. The gradient with respect to every weight in that layer is identical too. So training can never break the symmetry. The tiny-batch test from Step 1 would fail here for exactly this reason.

   • activations saturate near 0/1 immediately, or the first layer's gradient is exactly zero from step one — reinitialize using Chapter 13's Xavier/He schemes.
   • activations and gradients both look reasonable at initialization — continue to Step 4.

   Why: a network that starts in a degenerate state can't be rescued by any amount of training. That degenerate state means saturated activations, zero gradients, or broken symmetry. No learning rate, however well chosen, fixes it.

4. 4

   Manually inspect a handful of (input, label) pairs after every preprocessing step the pipeline runs. Confirm no information from validation or test data reached training.

   Example

   A refactor that re-sorted the feature table but not the label column: example 5's input now sits next to example 12's label. Every check in Steps 1–3 can pass perfectly, since the pipeline mechanics are fine. The data being fed in is simply wrong.

   • labels are shuffled, misaligned, or leaked information is present — fix the pipeline (see Section 6 of this chapter and Chapter 12's data-leakage material).
   • labels line up correctly and no leakage is found — continue to Step 5.

   Why: no amount of tuning can teach a network a relationship. The actual (input, label) pairs it's training on just don't contain that relationship.

5. 5

   Print the shape of the tensor entering and leaving every layer. Compare it against what the architecture is supposed to produce at that point.

   Example

   A layer meant to receive a (32, 256) batch instead receives (256, 32), a transpose bug upstream. Nothing crashes, because (256,32) broadcasts against the next layer's weights without error. The network trains on batch-of-256-features-times-32 nonsense instead of the intended 32-example batch. The loss curve looks exactly like an unrelated tuning problem.

   • a shape doesn't match what was intended (a broadcast filled in silently, a dimension got dropped or transposed) — fix the shape bug at its source.
   • every shape matches what's expected at every stage — continue to Step 6.

   Why: a shape mismatch that still runs without an error never announces itself. It just trains on the wrong numbers, and no hyperparameter fixes wrong numbers.

6. 6

   Log the gradient norm at every layer, every few steps. Look for two specific failure signatures: any layer sitting at exactly zero, and gradients shrinking toward zero from the output layer back toward the input.

   Example

   Layer 1's gradient norm reads exactly 0.0 at every one of 200 logged steps, while layer 4's gradient is a healthy 0.3. An exactly-zero reading that never budges is a broken connection: a detach, or a frozen parameter. A gradient that's small but not exactly zero, and shrinks layer by layer, is vanishing gradients instead.

   • some layer's gradient is always exactly zero, at every step — a disconnected or frozen path (a requires_grad flag left off, a non-differentiable operation in that path), not gradual vanishing.
   • gradients are nonzero everywhere but shrink sharply in the earliest layers — vanishing gradients (Chapter 13); revisit initialization and normalization.
   • gradients look reasonable at every layer — continue to Step 7.

   Why: "exactly zero, always" and "small, and shrinking with depth" point at two different bugs with two different fixes. The loss curve alone can't tell them apart. Only the gradient log can.

7. 7

   Watch for the exact step where loss first becomes NaN or Inf. Inspect what operation ran immediately before it. A log, a division, or an exponential are the usual suspects.

   Example

   A custom loss function computing \log(p) directly from a softmax output: once training pushes a predicted probability close enough to 0, \log(0)=-\infty appears. The very next gradient update turns every parameter into NaN.

   • loss becomes NaN or Inf partway through an otherwise normal-looking run — guard the offending operation (Section 8 of this chapter has the standard fixes).
   • no NaN or Inf appears anywhere, and the loss is still flat — the failure isn't one of these seven causes. Reconsider whether the model has enough capacity, or whether the task itself is set up correctly.

   Why: NaN and Inf are contagious. Once one appears anywhere in the computation, it propagates through every parameter it touches on the next update. So the run has to be caught and fixed at the exact operation that first produced it.