\frac{1}{\sqrt{d_k}}

The 1/√d_k scaling

Read "one over root d k". Attention divides its scores by the square root of the head width so they keep a spread of about 1, however wide the head is.

Suppose the entries of $\mathbf{q}$ and $\mathbf{k}$ are independent, with mean 0 and variance 1. Their dot product adds $d_k$ such products, so its variance is $d_k$ and its standard deviation is $\sqrt{d_k}$ . At $d_k = 64$ that is 8. Scores that far apart push softmax to its extremes: one weight near 1 and the rest near 0. There its slope, of the form $p(1-p)$ , is close to zero, so learning stalls.

Dividing by $\sqrt{d_k}$ brings the variance back to 1, keeping the weights spread out and the gradients useful.

Example

d_k = 64: \quad \operatorname{Var}(\mathbf{q}^\top \mathbf{k}) = 64 \;\Longrightarrow\; \operatorname{Var}\!\left(\tfrac{\mathbf{q}^\top \mathbf{k}}{8}\right) = \tfrac{64}{64} = 1

Dividing by √64 = 8 divides the variance by 8² = 64.

Where the course uses it

4.3 Logits, Softmax, and Cross-Entropy · Module 4, How a Model Learns From Datafirst use
6.1 The Question Attention Answers · Module 6, Attention
6.2 Scaled Dot-Product Attention, by Hand · Module 6, Attention
6.3 Why Divide by the Square Root of the Key Width · Module 6, Attention
8.1 The 2017 Encoder-Decoder and Cross-Attention · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
8.4 Decoder-Only: Predicting the Next Token · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
12.2 The KV Cache · Module 12, Generation
15.2 Stretching RoPE, and ALiBi · Module 15, The LLaMA Block
16.5 Logit Soft-Capping and QK-Norm · Module 16, Efficient and Long Attention

Related terms

Var — varianceRead "the variance of X". It measures how spread out a random quantity is around its average. Its square root is the standard deviation.
SoftmaxRead "softmax of z". It turns any list of numbers into probabilities: every output is positive and they add up to 1. Bigger inputs get bigger shares.
d_k, d_v — the width of one headRead "d k" and "d v". They are the number of entries in one attention head's key and value vectors, usually d_model divided by the number of heads.