query, key, value

The three vectors attention derives from each token -- what it's looking for, what it offers to be matched against, and what it contributes if matched.

For each token, three learned linear projections of its embedding produce a query vector (what this position is looking for), a key vector (what this position offers to be matched against), and a value vector (what this position contributes to the output if it is attended to). Attention compares each query to every key to compute weights, then combines values using those weights.

Examples

When "it" attends to "animal" in a sentence, "it"'s query vector is compared against "animal"'s key vector to produce the attention weight, and "animal"'s value vector is what actually gets mixed into "it"'s new representation.