attention (in a Transformer)

A mechanism letting each token look at every other token in context and weight how much each one matters to it.

Given a query vector, a set of key vectors, and a set of value vectors, attention computes $\text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$ : a similarity score between the query and each key, turned into weights by softmax, used to form a weighted sum of the values. Self-attention uses the same sequence for queries, keys, and values, so every token can attend to every other token in its context.

Examples

In "the animal didn't cross the street because it was tired," attention lets "it" weight "animal" heavily when forming its contextual meaning.