d_k, d_v, d_{\text{head}}

d_k, d_v — the width of one head

Read "d k" and "d v". They are the number of entries in one attention head's key and value vectors, usually d_model divided by the number of heads.

With $d_{\text{model}} = 512$ and $h = 8$ heads, $d_k = d_v = 512 / 8 = 64$ . Splitting the width this way means multi-head attention costs about the same as one full-width head, while each head can learn to focus on a different kind of relationship.

$d_k$ is the width of queries and keys, which must match because they are dotted together. $d_v$ is the width of the values. They are usually equal but do not have to be. $d_{\text{head}}$ is another name for the same per-head width.

Example

d_k = \frac{d_{\text{model}}}{h} = \frac{512}{8} = 64

The original Transformer's sizes.

Where the course uses it

1.2 Matrix Multiplication, Done by Hand · Module 1, Shapes and Linear Algebrafirst use
1.6 Linear Projections: One Vector Into Three · Module 1, Shapes and Linear Algebra
4.3 Logits, Softmax, and Cross-Entropy · Module 4, How a Model Learns From Data
6.1 The Question Attention Answers · Module 6, Attention
6.2 Scaled Dot-Product Attention, by Hand · Module 6, Attention
6.3 Why Divide by the Square Root of the Key Width · Module 6, Attention
6.6 Multi-Head Attention · Module 6, Attention
8.1 The 2017 Encoder-Decoder and Cross-Attention · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
8.4 Decoder-Only: Predicting the Next Token · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
12.2 The KV Cache · Module 12, Generation
15.2 Stretching RoPE, and ALiBi · Module 15, The LLaMA Block
15.5 Grouped-Query Attention · Module 15, The LLaMA Block
16.5 Logit Soft-Capping and QK-Norm · Module 16, Efficient and Long Attention
18.1 Linear Attention · Module 18, Beyond Attention: Linear Attention and Hybrids
23.1 The Memory Wall · Module 23, Serving
26.1 Connecting Vision to Language · Module 26, Vision-Language Models

Related terms

d_model — the model's widthRead "d model". It is how many numbers represent one token inside the network: 768 for GPT-2 small, 4096 for LLaMA-3 8B.
Query, key, valueThree learned versions of each token's vector. The query is what the token is looking for, the key is what it offers to others, and the value is what it passes on. W^Q is read "W Q", the matrix that makes queries.
The 1/√d_k scalingRead "one over root d k". Attention divides its scores by the square root of the head width so they keep a spread of about 1, however wide the head is.