\mathcal{L}

𝓛 — the loss

Read "L" or "the loss". The curly L is the single number that measures how wrong the model is. Training adjusts the weights to make it small.

$\mathcal{L}(\theta)$ is a function of the model's parameters $\theta$ : plug in the current weights and get back a score for how wrong the model is on the training data. Training runs gradient descent on this number, repeatedly nudging the weights in the direction that lowers it.

A subscript names which loss: $\mathcal{L}_{\text{NLL}}$ (negative log-likelihood), $\mathcal{L}_{\text{CTC}}$ (a loss for speech alignment), $\mathcal{L}_{\text{aux}}$ (the extra balancing term in mixture-of-experts). A model's total loss is often a weighted sum of several of these.

Example

\mathcal{L} = -\log 0.8 \approx 0.22

If the model gave the correct next token probability 0.8, the negative log-likelihood for that token is about 0.22. Probability 1 would give a loss of 0.

Where the course uses it

0.3 What You Need, and How the Course Is Organized · Module 0, Start Here: What an LLM Doesfirst use
1.1 Scalars, Vectors, Matrices, and Tensors · Module 1, Shapes and Linear Algebra
2.2 Partial Derivatives and the Gradient · Module 2, Calculus and Optimization
2.4 Gradient Descent · Module 2, Calculus and Optimization
2.5 Backpropagation · Module 2, Calculus and Optimization
3.4 Maximum Likelihood · Module 3, Probability and Information
4.3 Logits, Softmax, and Cross-Entropy · Module 4, How a Model Learns From Data
8.2 Encoder-Only: BERT and Masked Language Modeling · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
8.4 Decoder-Only: Predicting the Next Token · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
10.5 Scaling Laws: How Big, and on How Much Text · Module 10, Pretraining
17.4 Router Collapse and the Load-Balancing Loss · Module 17, Mixture of Experts
20.4 Distillation · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
21.3 PPO · Module 21, Preference Learning: RL, RLHF and DPO
21.4 Reward Models From Preferences · Module 21, Preference Learning: RL, RLHF and DPO
21.6 Direct Preference Optimization · Module 21, Preference Learning: RL, RLHF and DPO
25.3 Contrastive Learning: CLIP and SigLIP · Module 25, Vision Transformers and Contrastive Learning
26.4 Training on Images and Text Together · Module 26, Vision-Language Models
27.3 The CTC Objective and Its Recursion · Module 27, Recognition: OCR and Speech With CTC

Related terms

LogarithmRead "log of x". It undoes the exponential: log(eᶻ) = z. It turns products into sums, which is why loss functions use it.
θ — the model's parameters (or an angle)Read "theta". Usually it stands for every learnable weight in the model at once. In some sections it is an angle or a setting instead; the sentence around it tells you which.
∇ — gradientRead "nabla" or "the gradient of". It is the list of every partial derivative at once, and it points in the direction that increases the function fastest.