Attention mechanisms, one story
Queries, keys, and values, scoring functions, the scaling factor, and multi-head attention -- one mechanism, built up section by section.
Read this cluster as a single build, not a menu -- there's no situation in this course that calls for additive attention over scaled dot-product attention today, the way there might be a real choice between two activation functions. Start from the query/key/value vocabulary: a query is compared against every key to produce weights, which then combine the value vectors into one output. The scoring function that produces those weights can be as simple as a raw dot product or as flexible as a small feed-forward network (additive attention) -- scaled dot-product attention is the one that won on cost and performance, provided the score is divided by \sqrt{d_k} first. Multi-head attention is the last step: running several scaled dot-product computations in parallel on different learned projections, so a single layer can track several different relationships between tokens at once.
The vocabulary
Section
Queries, keys, and values
Scoring functions
Section
Scaled dot-product attention
Take query and key vectors q, k \in \mathbb{R}^{d_k} with independent, zero-mean, unit-variance entries. Their dot product q \cdot k then has variance d_k. Dividing the score by \sqrt{d_k}, \operatorname{score}(q, k) = \frac{q \cdot k}{\sqrt{d_k}}, restores unit variance regardless of d_k. This keeps the softmax over attention scores away from the saturated, near-zero-gradient region that a growing d_k would otherwise push it into.
Why the score is divided by \sqrt{d_k} -- restoring unit variance regardless of dimension.
Putting several heads to work
Example
Section