RoPE rotation matrix
Read "R theta m". It rotates a query or key vector by an angle that grows with the token's position m, so position is stored as an amount of rotation.
RoPE (rotary position embedding) splits the vector into pairs of numbers and rotates each pair, like a point on a plane, by the angle . Here is the position and is that pair's frequency. Fast-turning pairs distinguish nearby positions; slow-turning pairs distinguish distant ones.
Two rotations combine neatly: . So the score between a query at position and a key at position depends only on the distance . Each token is rotated by its own absolute position, yet attention only sees relative positions, and nothing is added to the embeddings.
Example
The rotation for one pair of numbers at position m.
Where the course uses it
Related terms
- θ — the model's parameters (or an angle)Read "theta". Usually it stands for every learnable weight in the model at once. In some sections it is an angle or a setting instead; the sentence around it tells you which.
- Orthogonal matrixA matrix that only rotates (or mirrors) vectors: it keeps every length and every angle, and its transpose undoes it. R^⊤R = I says exactly that.
- Scaled dot-product attentionEach token scores every other token, turns the scores into weights that add up to 1, and takes the weighted average of their value vectors.