KV cache

Storing each token's already-computed key and value vectors so generating the next token doesn't recompute attention over the whole sequence from scratch.

The KV cache stores the key and value vectors computed for every token already processed, so that generating the next token only requires computing that one new token's query, key, and value and attending against the cached keys/values -- turning what would be a full recomputation at every step into an incremental one.

Examples

Generating token 500 of a response with a KV cache means computing just that one new token's query against the 499 already-cached keys and values, not recomputing attention over all 500 tokens from scratch.