The optimizer family, one decision chain

Every optimizer in this course computes the same thing, a direction to step in and a size to step by, and differs only in how it refines those two numbers. Start from plain gradient descent's update rule and ask three questions in order. First: is a gradient computed over the full dataset too expensive per step? If so, move to stochastic gradient descent, which estimates the gradient from one minibatch instead. Second: does the loss surface have narrow, curved valleys that plain SGD zig-zags across? Momentum smooths that path by carrying a fraction of the previous step forward. Third: do different parameters need different effective step sizes -- some directions steep, others nearly flat? Adam adapts a per-parameter learning rate from each parameter's own recent gradient history, and is the default starting point for training a new network today. A learning-rate schedule and warmup sit on top of whichever of these is chosen, shrinking (or briefly growing, then shrinking) the step size over the course of training rather than fixing it once at the start.