gradient descent

An algorithm that repeatedly steps opposite the gradient to hunt for a local minimum.

Steepest ascent points along $\nabla f$ ; steepest descent, by the same logic run backward, points along $-\nabla f$ . Gradient descent turns that fact into a repeatable procedure: take a small step opposite the gradient, $\mathbf{x}_{n+1}=\mathbf{x}_n-\alpha\nabla f(\mathbf{x}_n)$ , recompute the gradient at the new point, and step again. The learning rate $\alpha$ trades off two failure modes: too small and convergence crawls; too large and a step can overshoot the minimum entirely and diverge.

Examples

Running gradient descent on $f(x,y)=x^2+2y^2$ from $(2,1)$ with $\alpha=0.3$ gives $(2,1)\to(0.8,-0.2)\to(0.32,0.04)\to(0.128,-0.008)$ .