\odot

⊙ — element-wise product

Read "a times b, element-wise". Multiply two same-shaped vectors or matrices position by position. This is not matrix multiplication.

$(\mathbf{a} \odot \mathbf{b})_i = a_i b_i$ . The output has the same shape as the inputs. In NumPy and PyTorch it is a * b, while matrix multiplication is a @ b. It is also called the Hadamard product.

It appears wherever each feature gets its own scale: the learned gain $\boldsymbol{\gamma}$ in LayerNorm and RMSNorm, and the gate in SwiGLU, where one branch scales the other entry by entry. Some texts write $\otimes$ for the same thing, though $\otimes$ can also mean other kinds of product.

Example

[1, 2, 3] \odot [4, 5, 6] = [4, 10, 18]

First times first, second times second, third times third.

Where the course uses it

1.2 Matrix Multiplication, Done by Hand · Module 1, Shapes and Linear Algebrafirst use
1.3 The Matrix Toolkit · Module 1, Shapes and Linear Algebra
7.3 LayerNorm · Module 7, The Block and the Full Model
15.3 RMSNorm · Module 15, The LLaMA Block
15.4 SwiGLU · Module 15, The LLaMA Block

Related terms

LayerNorm — layer normalizationRead "layer norm". It shifts and rescales each token's vector to have mean 0 and variance 1, then applies a learned scale and shift.
GELU, ReLU, Swish — activation functionsThe nonlinearity applied inside the feed-forward block. ReLU sets negative numbers to zero; GELU and Swish do almost the same but with a smooth curve.
Matrix multiplicationTwo matrices written side by side, like AB, are multiplied. Each entry of the result is a row of A dotted with a column of B. An (m×n) times an (n×p) gives an (m×p).