Recipe
Run one Adam update step
Use when: you have a gradient g_k at step k and Adam's running state m_{k-1}, v_{k-1} from the previous step (both 0 at k=0)
Goal: the updated parameter w_{k+1} and the running state to carry into step k+1
Taught in Adaptive methods: Adagrad, RMSProp, and Adam
Quick recipe
-
1
Compute m_k = \beta_1 m_{k-1} + (1-\beta_1)g_k.
Example
For f(w)=\frac12(w-5)^2 (so g=w-5) at w_0=0: g_1 = 0-5=-5. With \beta_1=0.9: m_1 = 0.9(0) + 0.1(-5) = -0.5.
Why: averaging the gradient itself, not just its square, smooths out direction the same way momentum does.
-
2
Compute v_k = \beta_2 v_{k-1} + (1-\beta_2)g_k^2.
Example
With \beta_2=0.999: v_1 = 0.999(0) + 0.001(-5)^2 = 0.025.
Why: this tracks how LARGE the gradient has typically been, regardless of sign, to scale the step per parameter.
-
3
Compute \hat m_k = m_k/(1-\beta_1^k) and \hat v_k = v_k/(1-\beta_2^k).
Example
At k=1: \hat m_1 = -0.5/(1-0.9) = -5. \hat v_1 = 0.025/(1-0.999)=25.
Why: m_0=v_0=0 biases the raw averages toward zero for the first several steps; this correction removes exactly that bias.
-
4
Compute w_{k+1} = w_k - \dfrac{\alpha}{\sqrt{\hat v_k}+\varepsilon}\,\hat m_k.
Example
With \alpha=1, \varepsilon\approx0: w_1 = 0 - \dfrac{1}{\sqrt{25}}(-5) = 0+1 = 1.
Why: dividing by \sqrt{\hat v_k} is exactly Adagrad/RMSProp's per-parameter scaling, applied to the smoothed gradient \hat m_k instead of the raw one.
Common mistakes
- Forgetting bias correction — the raw m_k,v_k are especially wrong in the first few steps, right when the correction matters most.
- Dividing by \sqrt{v_k} (the SECOND moment) but multiplying by g_k instead of m_k (the FIRST moment) — Adam scales the smoothed gradient, not the raw one.
- Reusing one \varepsilon inside the square root (\sqrt{\hat v_k+\varepsilon}) instead of adding it after (\sqrt{\hat v_k}+\varepsilon) — both prevent division by zero, but only the standard form matches how most implementations report it.