learning rate

The learning rate (or step size) \alpha>0 scales how far an iterative method moves along its chosen search direction at each step: x^{(k+1)}=x^{(k)}+\alpha\, d^{(k)}. Too large a value can overshoot a minimum or diverge; too small a value converges reliably but slowly — choosing \alpha well (fixed, decayed, or found by a line search) is central to every descent method's practical performance.