Recipe
Train a network with the training loop
Use when: you have a network, a chosen loss function, and a training dataset already split into batches, and need to actually run gradient-based training rather than describe one forward pass in the abstract
Goal: a set of trained parameters, produced by repeated forward and backward passes over the whole dataset
Taught in Putting it together: the training loop
Quick recipe
-
1
For epoch =1,2,\dots,E: run every step below once per epoch.
Why: One pass through the data rarely makes the loss bottom out. Repeating the whole pass is what "training" means, as opposed to just one forward pass.
-
2
Split the training set into batches (the next section makes this precise). Process them one at a time within the epoch.
Example
A dataset of N=240 examples with batch size B=16 makes m=240/16=15 batches. This epoch runs 15 iterations before moving to the next epoch.
Why: Chapter 9's minibatch gradient descent already showed that a gradient estimated from one batch is cheap to compute every step. It's also unbiased on average. This is that same choice, now driving a whole network instead of one linear model.
-
3
Run the current batch's inputs through the network, using the parameters exactly as they stand right now, to get predictions \hat y.
Why: Every later phase needs \hat y. Nothing about the loss or the gradient can be computed before the network has actually produced an output.
-
4
Compare \hat y to the batch's true targets with this problem's loss function. This produces one number L^{(t)}.
Why: The loss is the training signal (Chapter 7). It's the single quantity every later phase tries to make smaller.
-
5
Apply backpropagation to L^{(t)} to get \nabla_\theta L^{(t)}, the gradient with respect to every parameter.
Why: Backpropagation (Chapter 10) is the tool for turning one scalar loss into every parameter's own partial derivative. It does this in a single backward walk.
-
6
Move every parameter a small step against its own gradient: \theta \leftarrow \theta - \eta\nabla_\theta L^{(t)}.
Why: This step is what actually makes the loss tend to decrease. Moving against the gradient means moving in the direction the loss falls fastest, locally (Chapter 9). The rest of this chapter builds real practice on top of this same six-step skeleton, without changing it. Later sections cover: reading the resulting curve (Section 3), checking generalization (Section 4), saving progress and knowing when to stop (Section 5), shrinking the step over time (Section 6), and guarding this exact update against an oversized gradient (Section 7).
Common mistakes
- Computing a new batch's gradient without first clearing the previous batch's gradient. On a real computer, a training program that accumulates gradients by default (Section 9 shows exactly where this line goes in PyTorch) silently combines gradients from unrelated batches. This creates one oversized update if the reset is skipped.
- Applying the update before the backward pass has produced a gradient for every parameter, instead of after. An update needs the WHOLE gradient, not a partial one computed from an unfinished backward walk.
- Reporting an epoch count when the actual quantity tracked is the iteration count (how many updates have happened), or vice versa. Section 2 makes this precise.
- Judging a run only by its training loss, never checking a held-out set (Section 4). A model that has simply memorized the training data can look just as good as one that has actually learned the pattern.