Logarithm
Read "log of x". It undoes the exponential: log(eᶻ) = z. It turns products into sums, which is why loss functions use it.
. A sentence's probability is a product of hundreds of small numbers. Multiplied directly in floating point, the result gets so small it rounds to zero (called underflow). Taking the log turns the product into a sum of moderate negative numbers, which computers handle easily.
is the natural log, the log to base . In machine learning, almost always means as well. A probability is at most 1, so its log is zero or negative. That is why losses put a minus sign in front: it makes the loss a positive number to minimize.
Example
Natural log. A product of probabilities becomes a sum of their logs.
Where the course uses it
- 0.3 What You Need, and How the Course Is Organized · Module 0, Start Here: What an LLM Doesfirst use
- 2.1 Derivatives: Slopes and the Rules for Finding Them · Module 2, Calculus and Optimization
- 3.4 Maximum Likelihood · Module 3, Probability and Information
- 3.5 Information: Entropy, Cross-Entropy, KL, Perplexity · Module 3, Probability and Information
- 4.3 Logits, Softmax, and Cross-Entropy · Module 4, How a Model Learns From Data
- 8.2 Encoder-Only: BERT and Masked Language Modeling · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 8.4 Decoder-Only: Predicting the Next Token · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 13.1 Perplexity and Its Limits · Module 13, Evaluation
- 20.1 Supervised Instruction Tuning · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
- 21.2 Policy Gradients · Module 21, Preference Learning: RL, RLHF and DPO
- 21.4 Reward Models From Preferences · Module 21, Preference Learning: RL, RLHF and DPO
- 21.5 RLHF End to End · Module 21, Preference Learning: RL, RLHF and DPO
- 21.6 Direct Preference Optimization · Module 21, Preference Learning: RL, RLHF and DPO
- 25.3 Contrastive Learning: CLIP and SigLIP · Module 25, Vision Transformers and Contrastive Learning
- 26.4 Training on Images and Text Together · Module 26, Vision-Language Models
- 27.3 The CTC Objective and Its Recursion · Module 27, Recognition: OCR and Speech With CTC
Related terms
- The exponential functionRead "e to the z" (also written exp(z)). It raises e ≈ 2.718 to the power z. The result is always positive: e⁰ = 1, and it shrinks toward 0 as z becomes very negative.
- 𝓛 — the lossRead "L" or "the loss". The curly L is the single number that measures how wrong the model is. Training adjusts the weights to make it small.
- ∏ — productRead "the product of". The big pi works like ∑, but it multiplies the terms instead of adding them.