Transformers & Math
η\eta

η — the learning rate

Read "eta". The learning rate: how big a step gradient descent takes against the gradient on each update.

In θ←θ−η ∇θL\theta \leftarrow \theta - \eta \, \nabla_\theta \mathcal{L}, the gradient says which way to move and η\eta says how far. Too small and training crawls; too large and every step overshoots. Large models do not hold it fixed: section 10.2 raises it during a warmup and then lowers it along a schedule.

Example

θ=0,  ∇=−6,  η=0.1  ⟹  θ=0.6\theta = 0, \; \nabla = -6, \; \eta = 0.1 \;\Longrightarrow\; \theta = 0.6

One step of gradient descent on (θ − 3)², from section 2.5.

Where the course uses it