Recipe

Run one cycle of coordinate descent

Use when: you're minimizing a multivariable function by updating one variable at a time, holding the others fixed

Goal: an updated point after every coordinate has been minimized once, plus a check for whether that's already the exact minimizer

Taught in Coordinate descent

Quick recipe

Pick the next coordinate
Minimize along that one coordinate
Move to the next coordinate and repeat
Check convergence once every coordinate has updated

1
Go through the variables in order $x_1,\dots,x_n$ . Only the current coordinate $x_i$ changes this step — every other coordinate stays fixed at its current value.

Example

On $f(x,y)=(x-2)^2+(y-5)^2$ from $(0,0)$ , the cycle starts with $x$ : hold $y=0$ fixed.

Why: fixing every other coordinate is what turns an $n$ -variable problem into a sequence of trivial one-variable minimizations.
2
Solve $x_i\leftarrow\operatorname*{arg\,min}_{x_i}f(x_1,\dots,x_i,\dots,x_n)$ , treating every other coordinate as a fixed constant.

Example

Minimize $(x-2)^2+(0-5)^2$ over $x$ : $\dfrac{\partial}{\partial x}=2(x-2)=0\Rightarrow x=2$ , new point $(2,0)$ .

Why: this is Chapter 2's FONC applied to a single-variable slice of $f$ — nothing new, just restricted to one coordinate.
3
Update the point with the new coordinate value, move to the next variable, and repeat Step 2. Hold everything else, including the coordinate just updated, fixed at its latest value.

Example

With $x=2$ now fixed, minimize $(2-2)^2+(y-5)^2$ over $y$ : $y=5$ , new point $(2,5)$ .

Why: using the just-updated value, not the original one, is what makes this a genuine cycle rather than $n$ independent problems solved in parallel.
4
After one full cycle, check whether $f$ 's Hessian is diagonal — no cross terms coupling the variables.

Example

For $f(x,y)=(x+y)^2+0.01(x-y)^2$ from $(4,-2)$ , cycle 1 reaches $f\approx0.1522$ but cycle 6 has only reached $f\approx0.1020$ — this Hessian isn't diagonal.
- separable objective (diagonal Hessian) — the cycle just completed already reached the exact minimizer; stop.
- non-separable objective (off-diagonal entries) — the point has improved but hasn't converged; return to Step 1 for another full cycle.
Why: a diagonal Hessian means minimizing over one coordinate never disturbs what minimizing over another already found, so this check alone tells you whether one cycle was ever going to be enough.

Common mistakes

Updating two coordinates before re-minimizing, instead of one at a time in sequence.
Using a coordinate's OLD value from before this cycle instead of the value just updated earlier in the same cycle.
Assuming one cycle always finds the minimum — only guaranteed when the Hessian is diagonal.
Mistaking a big drop in $f$ after the first cycle for convergence, on a non-separable objective where progress can crawl for many more cycles.