\langle \mathbf{u}, \mathbf{v} \rangle

Dot product (inner product)

Read "the dot product of u and v". Multiply the two vectors entry by entry and add the results. You get one number, which is large when the vectors point the same way.

$\langle \mathbf{u}, \mathbf{v}\rangle = \sum_i u_i v_i$ . It also equals $\|\mathbf{u}\|\,\|\mathbf{v}\|\cos\theta$ , where $\theta$ is the angle between the vectors. The second form is why attention uses it: the dot product is large and positive when two vectors point the same way, near zero when they are perpendicular, and negative when they point apart.

Angle brackets $\langle \mathbf{u}, \mathbf{v}\rangle$ , $\mathbf{u}^\top\mathbf{v}$ and $\mathbf{u}\cdot\mathbf{v}$ are three ways of writing the same operation.

Example

\langle [1, 2, 3],\, [4, 0, -1] \rangle = 1{\cdot}4 + 2{\cdot}0 + 3{\cdot}(-1) = 1

Only slightly positive compared with the vectors' lengths (about 3.7 and 4.1), so they are close to perpendicular.

Where the course uses it

1.4 Dot Products, Length, and Angle · Module 1, Shapes and Linear Algebrafirst use
15.1 Rotary Position Embedding (RoPE) · Module 15, The LLaMA Block

Related terms

TransposeRead "A transpose". The raised T flips a matrix so its rows become its columns. On a vector it turns a column into a row.
Norm — the length of a vectorRead "the norm of x". Double bars mean the length of a vector: square each entry, add them up, and take the square root.
∑ — sumRead "the sum of". The big sigma is a loop that adds: the counter starts at the value below it and stops at the value above it.