Skip to content
Free public beta until December 4, 2026: every course is open, no card needed. After that, the first chapter of each course stays free and full access is a one-time $39 ($19 for up to 100 founding members chosen for helpful feedback). Your progress is kept. How the beta works

Every normalization scheme, and when to use it

Batch, layer, and RMSNorm -- three ways to keep a deep network's activations in a stable range, sorted by what each one actually normalizes over.

The schemes differ in exactly one thing: which set of numbers they average over to compute a mean and variance. Batch normalization averages across the examples in a minibatch, which makes it depend on batch size and unreliable with a batch of one -- a poor fit for sequence models and small-batch fine-tuning, and it needs a running average to behave consistently at inference time. Layer normalization averages across a single example's own features instead, so it works identically whether the batch has one example or a thousand -- the classic default inside a Transformer block. RMSNorm drops the mean-centering step entirely and normalizes only by root-mean-square magnitude, trading a small amount of expressiveness for a real speedup -- the choice behind most modern large language models. None of these replace careful initialization; Xavier and He initialization set the starting point these schemes then keep in range as training proceeds.

Where the problem starts

See section →
A gradient that shrinks toward zero as it's propagated backward through many layers, stalling learning in early layers.
See section →
A gradient that grows uncontrollably large as it's propagated backward through many layers, destabilizing training.

Getting the start right

See section →
Giving every neuron in a layer a different starting weight, so gradient descent can make them specialize instead of updating them identically forever.
See section →
A weight-variance formula, \sigma_W^2=2/(n+m) (fan-in n, fan-out m), that keeps a layer's pre-activation variance close to its input's, matched to sigmoid and tanh.
See section →
A weight-variance formula, \sigma_W^2=2/n (fan-in n), that doubles Xavier's variance to compensate for ReLU zeroing about half its input's variance.

Keeping every later layer in range, and choosing one

See section →
A technique that normalizes a layer's inputs using statistics computed across the current minibatch.

Example

Normalizes across the examples in a minibatch -- depends on batch size, and needs a running average at inference.
See section →
A technique that normalizes a layer's inputs using statistics computed across that layer's own features, per example.

Example

Normalizes across one example's own features -- identical behavior whether the batch has 1 example or 256.

Term

RMSNorm

See section →
Layer normalization with the mean-subtraction step dropped: rescale by the root mean square, \text{RMS}(z)=\sqrt{\frac1d\sum z_k^2}, instead of by the standard deviation.

Example

Layer normalization with the mean-subtraction step dropped -- cheaper, and the norm inside most modern Transformer blocks.
Graph settings

Calculator