Machine learning barely uses any other framing of "training a model." Fitting almost every method in this course means the same two steps. First, choose a loss function. Second, make it as small as possible. Gradient descent, or one of its versions, is how that shrinking actually happens. It's used whenever no closed-form formula exists.
Formula
The Gradient Descent Update Rule
Recalled from Calculus III — Multivariable Calculus · Gradient descent — first taught there.
To search for a local minimum of a differentiable function f, starting from a point \mathbf{x}_0, repeat \mathbf{x}_{n+1} = \mathbf{x}_n - \alpha\,\nabla f(\mathbf{x}_n), where \alpha>0 is a fixed (or shrinking) learning rate. Each step moves from \mathbf{x}_n a small distance in the direction of steepest descent, -\nabla f(\mathbf{x}_n), and stops improving (to first order) once \nabla f(\mathbf{x}_n)\approx\mathbf{0}.
Imagine standing on a hillside in thick fog. You're trying to reach the lowest point of a valley, with no map. All you can feel is which way the ground slopes under your feet, right now. The sensible strategy is simple. Take a small step downhill. Feel the new slope. Take another small step. Repeat. The loss function is the shape of that hillside. It's how "wrong" the model currently is, measured everywhere it could stand. The gradient is just "which way is downhill from here." The learning rate is how big a step to take each time. Too big, and you might stride right over the low point and start climbing the other side. Too small, and it takes forever to get anywhere. A gentle, bowl-shaped hillside (convex) has one lowest point, no matter where you start. A hillside with several separate dips (non-convex) is different. Where you happen to start decides which dip you end up stuck in.
The function being minimized is called the loss function L(\theta). It measures how wrong the model's current predictions are, on average, as a function of its parameters \theta. Training a model means repeatedly applying the update rule above: \theta \leftarrow \theta - \alpha \nabla L(\theta). Here, \nabla L(\theta) is the loss's gradient, the direction of steepest increase. \alpha is the learning rate. The rule says: step in the opposite direction of the gradient, by a small amount. Keep doing this until the loss stops meaningfully decreasing. Stochastic gradient descent computes \nabla L from a small random batch of data per step, rather than the whole dataset. It's the practical default for every model in this course large enough that a full-dataset gradient, every step, would be too slow. Convexity determines what that stepping process can promise. A convex loss, like linear regression's squared error, has one global minimum. Gradient descent is guaranteed to find it. A non-convex loss, like a neural network's (briefly previewed in Chapter 16), can have many local minima. Here, gradient descent only promises to find some minimum, not necessarily the best one.
Gradient descent minimizing a loss surface
Watch the marker descend toward the minimum. Toggle momentum on and compare how many fewer steps it needs on this stretched, ill-conditioned bowl -- the same shape a poorly-scaled feature matrix produces.
Loading interactive…
Worked example
Reading a loss surface's shape as a training difficulty
-
1
Two features are on very different scales: one ranges 0-1, the other ranges 0-100,000. Without rescaling, is gradient descent on this dataset's loss more likely to converge quickly or to zig-zag slowly?
-
2
An unscaled feature with a much larger range stretches the loss surface into a long, narrow valley along that feature's coefficient axis — exactly the ill-conditioned shape the widget above shows gradient descent struggling with.
-
3\boxed{\text{Zig-zag slowly, unless the features are rescaled first}}
This is the concrete reason Chapter 4's feature scaling isn't just cosmetic — it changes the literal shape of the surface gradient descent has to descend.
Loading interactive…
Worked example
Taking one gradient descent step by hand
-
1
A loss function is L(\theta) = (\theta - 5)^2, so \nabla L(\theta) = 2(\theta - 5). Starting at \theta = 0 with learning rate \alpha = 0.1, what is \theta after one gradient descent update?
-
2
\nabla L(0) = 2(0 - 5) = -10. The update rule gives $\theta \leftarrow 0
- 0.1 \times (-10)$.
-
3\boxed{\theta = 1}
Stepping in the direction opposite the gradient moved \theta from 0 toward 5, the true minimum — exactly what "descent" means. Repeating this update, \theta keeps approaching 5, taking smaller steps as \nabla L itself shrinks near the minimum.
Loading interactive…
Dragging a loss surface from convex to non-convex
Drag the ripple slider up from 0. At 0 the bowl is convex -- one minimum, gradient descent always finds it. Past about 1.5, extra local minima appear, and where descent starts decides which one it lands in.
Loading interactive…
Where this goes next
Every method from Chapter 6 (linear regression) through Chapter 16 (the
neural-network bridge) fits its parameters the same way. It makes some
loss function as small as possible. It does this using gradient
descent, or a closed-form shortcut that reaches the exact same answer.
Chapter 10's SVM and Chapter 11's Lagrangian threshold tuning work
differently. Both lean on optimization's tools for constrained
problems, not on gradient descent alone.
Check your understanding (counts toward your progress)
Stochastic gradient descent computes the gradient using a small random batch instead of the full dataset. If a dataset has 10,000 samples and a batch size of 200 is used, how many batches make up one full pass (epoch) through the data?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
One epoch covers every sample exactly once, so the number of batches per epoch is the dataset size divided by the batch size -- 10000/200=50.
Correct answer
Also accepted:
Why
Why does a convex loss function matter for gradient descent's guarantees?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Convexity guarantees WHICH minimum is found, not that training is always fast — an ill-conditioned convex problem can still be slow.
A convex loss's minimum value depends on the problem; convexity says nothing about that value being zero.
Convexity is a property of the loss function itself and applies to regression losses (like squared error) just as much as classification losses.
Why
A convex loss has a single global minimum, so gradient descent's local-improvement steps are guaranteed to converge to the actual best solution, not get stuck at a worse one.
Two features are on very different numeric scales, left unscaled. What effect does this typically have on gradient descent?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
A smaller learning rate can reduce the zig-zag but doesn't remove the underlying ill-conditioning caused by mismatched scales.
Feature scale affects the loss surface's shape for both supervised and unsupervised methods that use gradient-based fitting.
Ill-conditioning slows convergence; it does not generally prevent gradient descent from converging at all, given enough iterations.
Why
Unscaled features on very different ranges stretch the loss surface into a narrow valley, which is exactly the shape gradient descent zig-zags slowly across.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Optimization
Three examples pull a weight w toward targets c_1=4, c_2=6, c_3=1, with per-example gradient g_i(w)=w-c_i. At w^{(0)}=0, find g_2(w^{(0)}).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
g_2(0) = 0 - 6 = -6.
Correct answer
Also accepted:
Why
From Optimization
A different 3-example set has targets c_1=10, c_2=2, c_3=6. Using \alpha=0.05 from w^{(0)}=0 and example 1 only, find w^{(1)}.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
g_1(0) = 0-10=-10, so w1 = 0 - 0.05(-10) = 0.5.
Correct answer
Also accepted:
Why
Quick review
Practice from earlier coursework, pulled in for this topic.
From Optimization
Jensen's inequality's k-point form reduces exactly to Section 4's original definition when:
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
The two-point case, k=2, is literally Section 4's own convexity inequality, restated with subscripted points and weights.
From Optimization
f is convex on C if, for every x,y\in C and \theta\in[0,1]:
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That inequality direction defines CONCAVE, the mirror image of convex.
Why
The chord's predicted value never undershoots the function's actual value at the blended point -- the reverse inequality is concavity's own definition.