Section 5 ended on a promise. Instead of forcing a whole sequence through one repeatedly-rewritten summary, keep every position's own copy available. Let whatever needs information decide, per query, which positions to draw from. This section makes that promise concrete, with numbers small enough to check by hand. It's a complete, 3-token attention computation, worked start to finish. Every later chapter in this course reuses it, without re-deriving it.
Three vectors, three jobs. The fastest way to keep them apart is a librarian scene. A query is a request slip: "I'm looking for something like this." A key is the subject line printed on a catalog card. It's used only to be compared against the request -- never handed over directly. A value is what's actually sitting on the shelf that card points to. It's the thing that gets retrieved, in some proportion, once a card is judged relevant. Every one of these is an ordinary vector: a fixed-length list of numbers. Nothing is new about the object -- only about the role it's playing this time.
The recipe writes that scene down as arithmetic, in three steps. Step 1
compares the request against every card. q \cdot k_i is the ordinary
dot product (Chapter 2 covers this measurement in depth; for now, a
bigger number means a better match). Divide that by \sqrt{d_k}. d_k
(decoded: "size of k," how many numbers each query and key vector has)
sits under a square root. That's a fixed rescaling. It keeps the raw
scores from growing huge just because the vectors happen to be long.
(deep-learning chapter 20 derives exactly why, in full; this section
only needs that the division happens.) Step 2 runs those scores through
softmax. Softmax is a function that turns any list of numbers into
genuine percentages: all between 0 and 1, all adding up to 100%. That
produces attention weights a_i: "card 3 earned 66% of the total
match." Step 3 blends the shelf contents by those exact percentages.
\sum_i a_i v_i reads as "take 66% of value 3, plus smaller shares of
every other value, and add them together." It's a graded blend, weighted
toward the best match -- never a single card copied out whole.
Self-attention is the specific case this course actually uses everywhere. Instead of one separate request compared against a different shelf of cards, every single word in a sentence gets to be the requester. Each word compares itself against every other word in that same sentence -- including itself. Concretely: in "the trophy didn't fit because it was too big," the word "it" acts as a query. It can attend directly back to "trophy" acting as a key, pulling in "trophy"'s value to help decide what "it" refers to. That's the exact three-step recipe above -- just with every request, card, and shelf drawn from the one sentence itself.
Attention operates on three vectors per token. A query q is what this position is currently looking for. A key k is what this position offers, to be matched against. A value v is what this position actually contributes, if attended to. Given one query q and n key/value pairs (k_1,v_1),\ldots,(k_n,v_n), attention runs in three steps. (1) Score the query against every key. (2) Normalize the scores into a probability distribution, with softmax. (3) Combine the values, weighted by those normalized scores.
Scaled dot-product attention is the specific version every Transformer in this course uses. The score between query q and key k_i is their dot product, divided by \sqrt{d_k} (the shared size of every query and key vector): s_i = \frac{q \cdot k_i}{\sqrt{d_k}}, \qquad a_i = \operatorname{softmax}(s_1,\ldots,s_n)_i = \frac{e^{s_i}}{\sum_j e^{s_j}}, \qquad \operatorname{output} = \sum_i a_i v_i. a_i, the attention weight, is a number between 0 and 1, with \sum_i a_i = 1 -- exactly softmax's usual guarantee (Section 8). In self-attention, the mechanism every Transformer block in this course runs, every token in a sequence plays all three roles at once. Each position's own embedding produces its own query, key, and value. Every position attends back over the same sequence -- including, unless masked (Section 7), itself.
Formula
Scaled Dot-Product Attention Scaling
Recalled from Deep Learning / Neural Networks · Why the scaling factor? — first taught there.
Take query and key vectors q, k \in \mathbb{R}^{d_k} with independent, zero-mean, unit-variance entries. Their dot product q \cdot k then has variance d_k. Dividing the score by \sqrt{d_k}, \operatorname{score}(q, k) = \frac{q \cdot k}{\sqrt{d_k}}, restores unit variance regardless of d_k. This keeps the softmax over attention scores away from the saturated, near-zero-gradient region that a growing d_k would otherwise push it into.
Drag the query and watch every score, weight, and output recompute
Start with q at (0, 1) -- 'love,' attending over I, love, and NLP. Drag q toward k_3 (NLP, at (1,1)) until its weight is the largest of the three, then read the output value shift toward v(\text{NLP}) = 4.
Loading interactive…
The full 3-token self-attention matrix, from this section's own numbers
Click each row and confirm it matches this section's worked examples -- row 'love' should read close to (0.20, 0.40, 0.40), and every row's three weights should sum to 1.
Loading interactive…
Worked example
Basic: one query, three keys, the full pipeline
-
1
The sentence is "I love NLP," with q_{\text{love}}=(0,1), k_{\text{I}}=(1,0), k_{\text{love}}=(0,1), k_{\text{NLP}}=(1,1), so d_k=2. Compute the three scaled scores for query "love."
-
2
Raw dot products: q\cdot k_{\text{I}} = (0)(1)+(1)(0) = 0; q\cdot k_{\text{love}} = (0)(0)+(1)(1) = 1; q\cdot k_{\text{NLP}} = (0)(1)+(1)(1) = 1. Divide each by \sqrt{d_k}=\sqrt2\approx1.4142.
-
3
\boxed{s_{\text{I}}=0,\ s_{\text{love}}\approx0.7071,\ s_{\text{NLP}}\approx0.7071}
Worked example
Typical: finishing query 'love' to an output
-
1
Continuing the previous example, with values v_{\text{I}}=2, v_{\text{love}}=6, v_{\text{NLP}}=4, compute the softmax weights and the final attention output for query "love."
-
2
Softmax over (0,\ 0.7071,\ 0.7071): subtract the max (0.7071), giving (-0.7071,\ 0,\ 0), exponentiate to (0.4931,\ 1,\ 1), sum \approx2.4931, normalize: a_{\text{I}}\approx0.1978, a_{\text{love}}\approx0.4011, a_{\text{NLP}}\approx0.4011.
-
3
\operatorname{output} = 0.1978(2) + 0.4011(6) + 0.4011(4) \approx 0.3956+2.4066+1.6044.
-
4
\boxed{\operatorname{output}\approx4.41} -- closer to v_{\text{love}} and v_{\text{NLP}} (the two highest-weighted values) than to v_{\text{I}}=2, matching the widget above once q is set back to (0,1).
Worked example
Variation: a different query token, same three keys
-
1
Same sentence and keys, but now query "I": q_{\text{I}}=(1,0). Compute the scaled scores, softmax weights, and final output.
-
2
Raw dot products: q\cdot k_{\text{I}}=1, q\cdot k_{\text{love}}=0, q\cdot k_{\text{NLP}}=1. Scaled: s\approx(0.7071,\ 0,\ 0.7071).
-
3
Softmax: subtract the max, giving (0,\ -0.7071,\ 0), exponentiate to (1,\ 0.4931,\ 1), sum \approx2.4931, normalize: a_{\text{I}}\approx 0.4011, a_{\text{love}}\approx0.1978, a_{\text{NLP}}\approx0.4011.
-
4
\operatorname{output}=0.4011(2)+0.1978(6)+0.4011(4)\approx0.8022+1.1868 +1.6044\approx\boxed{3.59} -- a genuinely different output from query "love"'s 4.41, even though the keys and values never changed -- the query alone decides what gets attended to.
Worked example
Common trap: mixing up which vector does the comparing
-
1
A student scores each key by comparing the query against the values instead of the keys -- computing q \cdot v_i rather than q \cdot k_i. What breaks?
-
2
Keys exist specifically to be compared against a query; values exist specifically to be retrieved once a match is found. Scoring against values conflates the two jobs -- relevance would depend on the exact content retrieved, rather than on a separate description built for matching.
-
3
\boxed{\text{Scores must compare } q \text{ against } k_i\text{, never against } v_i} -- in this section's own numbers, v_{\text{I}}=2 is just a number, not a vector q could even take a dot product against in general; keys and values can have entirely different shapes, precisely because they do different jobs.
🧪 Computational Laboratory Run it
Reproduce query "love"'s attention output in NumPy
Run the exact three-key, three-value computation above in a few lines of NumPy and confirm the printed output matches 4.41.
Runs in your browser through Google Colab, nothing to install. Or download it and open it in Jupyter.
Where this goes next
Section 7 (the Transformer block) and every later chapter
Every remaining section in this course reuses exactly this q,k,v vocabulary, and this exact scaled dot-product formula. Section 7 wraps it in a full Transformer block. It also adds masking, so a decoder can't peek at future tokens. Chapter 1 onward never re-derives attention from scratch again. This section's three steps are the mechanism, verbatim, every time this course says "the model attends to."
Check your understanding (counts toward your progress)
In the query/key/value framework, what role does a key play?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That describes the value's role, not the key's -- values are what gets retrieved, keys are only for comparison.
That describes the query's role -- the request being made, not what's offered for matching.
There's no such running count in this framework -- keys exist purely to be compared against a query.
Why
A key is compared against a query to compute a relevance score; it is never handed over as the actual result -- that's the value's job.
Two different queries, q_A and q_B, attend over the identical set of keys and values. Is it possible for them to produce different attention outputs?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The query is exactly what determines each key's relevance score -- it has a direct effect on the output.
Attention is highly sensitive to all three inputs, including the query -- it isn't a fixed, input-independent output.
The keys and values can stay fixed; changing only the query is sufficient to change which ones get emphasized.
Why
The query alone decides how relevant each key is judged to be -- this section's own worked examples showed queries "I" and "love," using the identical keys and values, producing genuinely different outputs (3.59 versus 4.41).
This section's own two worked examples (queries "I" and "love") used the identical keys and values -- compare their outputs.
In self-attention specifically, where do a token's query, key, and value vectors come from?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That describes cross-attention (a decoder attending to a separate encoder), not self-attention specifically.
Keys and values are also learned per-token projections, not shared fixed constants.
Queries and keys are learned projections of the token's embedding, not hand-set hyperparameters.
Why
Self-attention derives queries, keys, and values all from the same input sequence -- every position attends back over that identical sequence, including itself.
"Self" in self-attention names where ALL THREE vectors come from.
What are the three steps of the attention recipe, in order?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
This scrambles the actual order and mixes up which object plays which role.
Keys and values aren't combined before scoring -- scoring happens first, against the query.
Values aren't scored against each other, and queries aren't normalized -- this misassigns every role.
Why
The three steps are score (query against every key), normalize (softmax turns scores into weights), combine (a weighted sum of the values using those weights).
Why does scaled dot-product attention divide each score by \sqrt{d_k} rather than using the raw dot product directly?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Softmax is still a separate, required step after the scaling -- scaling alone doesn't produce a probability distribution.
Scaling doesn't force equal weights -- it rescales the SPREAD of scores, but relevant differences still come through after softmax.
The scaling factor's job doesn't depend on the sign of the entries -- it addresses how the dot product's variance grows with dimension d_k generally.
Why
The raw dot product's variance grows with d_k; dividing by \sqrt{d_k} restores a roughly constant spread, keeping softmax away from the saturated, near-zero-gradient region a growing d_k would otherwise push it into.
This section's recalled formula card names exactly what the scaling factor restores about the score's variance.
What guarantee does softmax provide about the attention weights a_1,\ldots,a_n it produces?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Softmax weights are only equal when the input scores happen to be equal -- in general, they differ based on relevance.
A one-hot output only happens in the extreme limit -- ordinary softmax spreads weight across every candidate, in varying amounts.
Softmax outputs are always non-negative -- they can never be negative, by construction (each is e^{s_i} divided by a positive sum).
Why
Softmax guarantees a genuine probability distribution: every output strictly between 0 and 1 (in practice), summing to exactly 1 across all candidates.
Check your understanding (counts toward your progress)
Continuing the previous question (a1 = 0.731, so a2 = 0.269), with values v1 = 10 and v2 = 2, compute the final attention output. Round to 2 decimal places.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
\operatorname{output} = 0.731(10) + 0.269(2) = 7.31 + 0.538 \approx 7.85.
Correct answer
Also accepted:
Why
The output is the weighted sum: a1v1 + a2v2.
In this section's 3-token self-attention matrix, every row's three weights sum to what value?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Each row is one query's own softmax output over all candidate keys -- softmax always sums to 1, not to the number of columns.
This is a guaranteed property of softmax, true for every row identically, not a varying total.
Softmax outputs are non-negative and sum to 1 -- never 0, unless every candidate were somehow given zero weight, which softmax never does.
Why
Every row of an attention matrix is one query's own softmax distribution over the keys -- by softmax's own guarantee, each row always sums to exactly 1.
Two scaled scores are s1 = 1.0 and s2 = 0.0 (a two-key case). Compute softmax weight a1, the weight on the first key. Round to 3 decimal places.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Subtract the max (1.0): (0, -1.0); exponentiate: (1, 0.3679); sum \approx1.3679; a_1 = 1/1.3679\approx0.731.
Correct answer
Also accepted:
Why
Subtract the max score from both before exponentiating, for stability: (0, -1.0).
With q = (2, 0) and k = (3, 1) -- notice q's second component is zero -- compute the raw dot product q . k.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
q\cdot k = (2)(3) + (0)(1) = 6.
Correct answer
Also accepted:
Why
Multiply matching components and add: (2)(3) + (0)(1).
This section's own worked example computed query "love"'s attention output as approximately 4.41, using values 2, 6, and 4 for I, love, and NLP. Which statement about that output is correct?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The plain average of 2, 6, 4 is exactly 4.0, not 4.41 -- and the weights in this example were NOT all equal (love and NLP both outweighed I).
A weighted sum of the values is a genuine blend, not required to land on any one original value.
A weighted average can never exceed the largest input value (6 here) -- 4.41 sits comfortably below that, not above it.
Why
A weighted sum of values always lies between the smallest and largest value being combined -- 4.41 sits between 2 and 6, pulled toward the two more heavily-weighted values (6 and 4) and away from the lightly-weighted 2.
An attention output is a WEIGHTED average -- compare it to the plain, unweighted average of the same three values (which is exactly 4.0).
With q = (2, 0), k = (3, 1), and d_k = 2, compute the scaled score q.k / sqrt(d_k). Round to 3 decimal places.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
q\cdot k = 6; \sqrt{d_k}=\sqrt2\approx1.4142; 6/1.4142\approx4.243.
Correct answer
Also accepted:
Why
The raw dot product is 6 (from the previous question); divide by sqrt(2).
A student scores a query against keys using q\cdot v_i (the values) instead of q\cdot k_i (the keys). What's wrong with this?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Keys and values are defined to play distinct roles -- swapping them changes what the score is actually measuring.
Values can be vectors just as easily as scalars in general self-attention -- this section's own numeric example just happened to use scalar values for simplicity.
Softmax works on whatever numeric scores it's given -- the issue is conceptual (what the score measures), not a mathematical restriction on softmax itself.
Why
Keys and values are built for different jobs -- comparing against the values instead of the keys makes relevance depend on the exact content that would be retrieved, rather than on a purpose-built description meant for matching.
With q = (1, 1), k = (1, -1), and d_k = 2, compute the scaled score q.k / sqrt(d_k). Round to 3 decimal places.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
q\cdot k = (1)(1) + (1)(-1) = 0; 0/\sqrt2 = 0 -- the query and key are orthogonal, so their similarity score is exactly zero regardless of scaling.
Correct answer
Also accepted:
Why
Compute the raw dot product first: (1)(1) + (1)(-1).
Practice: Scoring, weighting, and combining -- the attention recipe by hand (5 problems · doesn’t affect your progress)
Which vector actually gets blended together to form the attention output?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
The final output is a weighted sum of the VALUES, using the softmax-normalized attention weights.
The values, weighted by the attention weights
With q = (1, 2) and k = (2, 1), compute the raw dot product q.k.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
q\cdot k = (1)(2) + (2)(1) = 4.
Correct answer
Also accepted:
Why
(1)(2) + (2)(1).
With d_k = 4, scale the raw dot product 8 by dividing by sqrt(d_k). Round to 3 decimals.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
\sqrt4=2; 8/2=4.0.
Correct answer
Also accepted:
Why
sqrt(4) = 2; divide 8 by 2.
Two scaled scores are s1 = 2.0, s2 = 0.0. Compute softmax weight a1 (round to 3 decimals).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Subtract the max: (0, -2.0); exponentiate: (1, 0.1353); sum \approx1.1353; a_1 = 1/1.1353 \approx0.881.
Correct answer
Also accepted:
Why
Subtract the max (2.0) from both first: (0, -2.0), then exponentiate and normalize.
Two different queries attend over the identical keys and values. Can they produce different outputs?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
The query determines every score against every key -- changing only the query, with keys and values fixed, is enough to change the output, exactly as this section's own two worked examples showed.
Yes -- the query alone determines each key's relevance, so a different query can shift the weights and the output