K_{\text{cache}}

The KV cache

Read "K cache" and "V cache". They store the keys and values of earlier tokens during generation, so each is computed once instead of again at every step.

During generation, the keys and values of earlier tokens never change; each step only adds one new row. Storing them cuts the attention work at step $t$ from $\mathcal{O}(t^2)$ to $\mathcal{O}(t)$ , so generating $T$ tokens costs $\mathcal{O}(T^2)$ instead of $\mathcal{O}(T^3)$ .

The cost is memory. The cache grows in proportion to context length and batch size, and for long contexts or large batches it can take more memory than the model's weights. Reducing it is the main motivation for grouped-query attention (GQA), multi-query attention (MQA) and paged attention.

Where the course uses it

12.2 The KV Cache · Module 12, Generationfirst use

Related terms

Big-O — how cost growsRead "order of" or "big O of". It describes how cost grows as the input gets bigger, ignoring constant factors. O(N²) means doubling N roughly quadruples the work.
Scaled dot-product attentionEach token scores every other token, turns the scores into weights that add up to 1, and takes the weighted average of their value vectors.
Superscript in parentheses — a labelRead x^(t) as "x at step t". A raised number or letter in parentheses is a label, such as a time step. It is never a power.