Cross-entropy — the training loss
The average surprise when the truth is p but you predict with q. Against a one-hot target it is minus the log of the probability given to the correct class.
Minimizing it is the same as maximizing likelihood (sections 3.4 and 3.5). It is the loss of every classifier and language model in the course.
Where the course uses it
Related terms
- Entropy — average surpriseThe average surprise of a distribution, H(p) = −Σ p log p: zero if one outcome is certain, largest when all outcomes are equally likely.
- KL — Kullback–Leibler divergenceHow different two probability distributions are: zero when they agree, larger the more they differ. It is not symmetric, so KL(P‖Q) and KL(Q‖P) differ.
- LogarithmRead "log of x". It undoes the exponential: log(eᶻ) = z. It turns products into sums, which is why loss functions use it.