Skip to content
Free public beta until December 4, 2026: every course is open, no card needed. After that, the first chapter of each course stays free and full access is a one-time $39 ($19 for up to 100 founding members chosen for helpful feedback). Your progress is kept. How the beta works

Recipe

Fit a model with gradient descent

Use when: you have a model with adjustable weights and a bias, a loss function, and a dataset, and you want to find weights that make the loss small

Goal: a full sequence of updates that drives the loss toward its lowest reachable value

Taught in Fitting the line with gradient descent

Quick recipe

  1. Make a prediction
  2. Compute the loss
  3. Compute the gradient
  4. Update every parameter
  5. Repeat
  1. 1

    Run every example through the current rule, \hat y_i = wx_i + b, using whatever w and b currently are.

    Example

    At w=-2, b=26: the five cars get predictions 22, 18, 14, 10, 6.

    Why: every later step needs to know how the rule is currently doing, and that starts with what it currently outputs.

  2. 2

    Compare every prediction to its true value and combine the misses into one number, using Chapter 7's loss.

    Example

    Residuals 2, 3, 3, 4, 3; squared 4, 9, 9, 16, 9; mean 47/5 = 9.4.

    Why: the loss is the single number the rest of this recipe is trying to make smaller — without it, "improve the rule" has no meaning to check against.

  3. 3

    Find how the loss changes with respect to every parameter — one partial derivative per weight, plus one for the bias, using Chapter 8's chain rule.

    Example

    \partial L/\partial w = -38.4, \partial L/\partial b = -6.

    • the gradient uses every training example — this is full-batch gradient descent, this section's own case.
    • the gradient uses a random subset of examples — this is minibatch or stochastic gradient descent, covered next in Section 6.

    Why: the update in the next step needs to know which direction increases the loss, for every parameter, before it can move against it.

  4. 4

    Move every parameter by the learning rate times its own gradient entry, subtracting, never adding.

    Example

    w \leftarrow -2 - 0.01(-38.4) = -1.616; b \leftarrow 26 - 0.01(-6) = 26.06.

    Why: subtracting a positive multiple of the gradient is exactly what moves every parameter toward lower loss, by Section 2's own argument.

  5. 5

    Go back to step 1 with the new w and b, and run the whole sequence again.

    Why: one step only ever makes a small improvement; reaching the bottom of the bowl takes many repeats of the same four moves.

Common mistakes

  • Forgetting to average the gradient over the batch, so the step behaves like a much larger, uncontrolled learning rate (Section 6 names this exactly).
  • Choosing a learning rate so large the loss grows instead of shrinking (Section 3's divergence).
  • Dropping the minus sign in the update, moving every parameter the wrong way (Section 2's own common trap).
  • Recomputing the loss without first recomputing the predictions from the new w and b — the loss always has to match the parameters it's reporting on.