gradient descent

An iterative method that repeatedly steps in the direction the objective decreases fastest.

Starting from an initial guess $x^{(0)}$ , gradient descent repeats the update $x^{(k+1)} = x^{(k)} - \alpha \nabla f(x^{(k)})$ , stepping a distance controlled by the learning rate $\alpha$ in the direction opposite the gradient — the direction $f$ decreases fastest locally, since $-\nabla f(x^{(k)})$ is exactly the steepest-descent direction at $x^{(k)}$ .