Skip to content
Free public beta until December 4, 2026: every course is open, no card needed. After that, the first chapter of each course stays free and full access is a one-time $39 ($19 for up to 100 founding members chosen for helpful feedback). Your progress is kept. How the beta works

self-attention

Attention where the queries, keys, and values all come from the same sequence.

Examples

  • In self-attention over the sentence "the cat sat", the word "sat" can attend directly back to "cat" regardless of distance between them.

See also

query, key, and value multi-head attention Transformer
Read the full section →
Graph settings

Calculator