The loss says how well a model predicts held-out text, not how useful it is: whether it answers correctly, writes working programs, follows instructions, or finds a fact on page 300. This module covers how models are actually measured and how far to trust the numbers. The habits of section 4.6 carry over: a benchmark score is a measurement, with noise, a scale, and ways to be gamed.

13.1 Perplexity and Its LimitsRun this section

Perplexity (section 3.5) is the cheapest evaluation: no labels needed, and within one model family it tracks capability well.

Its main limit is easy to miss. Perplexity is per token, so it depends on the tokenizer. Long tokens mean fewer, harder predictions for the same text, so per-token perplexity is higher even when the text is predicted equally well. The total negative log-likelihood of a text, its total surprise, does not depend on how it was cut up, so dividing it by characters or bytes gives bits per character or bits per byte, comparable across tokenizers:

\text{bits per byte} = \frac{1}{n_{\text{bytes}}} \sum_{t=1}^{T} -\log_2 P_\theta(w_t \mid w_{<t})

introduces 4 notations

Perplexity on web text also says little about skills rare in that text, such as multi-step arithmetic. And post-training (Modules 20 to 22) deliberately makes models less like average web text, so their perplexity can rise as they become more useful.

Browser lab

Perplexity changes with the tokenizer; bits per character much less so

The same simple model (a smoothed bigram model: each token predicted from the one before) is trained on the same text three times, with three BPE vocabularies of increasing size, and evaluated on the same unseen sentence. Section 13.1 predicts that perplexity per token moves a lot with the vocabulary, so it cannot rank the three, while bits per character, which normalizes by the text instead of the tokens, is the comparable measure.

A bigram model and a tiny corpus, chosen so the lab runs instantly; the bigram also sees more context per token when tokens are longer, so bits per character can legitimately differ between the rows. The point is which number is safe to compare, not which tokenizer is best.

13.2 BenchmarksRun this section

A benchmark is a fixed set of test questions with a scoring rule:

Multiple choice (MMLU, 57 academic subjects, is the best known). Scored either by asking for the letter, or by comparing the likelihood of each option. The two can differ noticeably, so a number means little without its method.
Exact match. A short answer must equal the reference after normalization (lower-casing, stripping punctuation); mathematics benchmarks such as GSM8K compare the final number.
Execution. For code, hidden tests run the model's program (HumanEval, MBPP). The standard measure is pass@k, the probability that at least one of $k$ samples passes, estimated without bias from $n \ge k$ samples of which $c$ pass:

\text{pass@}k = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}}

The exact prompt (wording, and examples shown first, few-shot prompting) and the answer extraction each move scores by several points, so comparisons are fair only with both fixed. Benchmarks also saturate: once the best models score near 100%, the field moves to harder ones.

13.3 Contamination

Contamination means the test questions or answers appeared in the pretraining data. Benchmarks are published, discussed and copied across the web, so a crawl contains many unless they are removed, and a contaminated model scores by recall instead of skill. Imperfect defences:

Decontaminate the training data by removing long overlaps (say 13 words) with benchmark items, using the shingle matching of section 10.1.
Test for memorization: compare published questions with freshly written ones, or check whether the model completes a question verbatim from its first half.
Use dynamic or private benchmarks, refreshed regularly, dated after the training cutoff, or never published.

A softer form: when a field tunes its methods against one benchmark for years, the methods overfit to it even if no model saw a test question.

13.4 Long-Context Evaluation

Accepting hundreds of thousands of tokens is not the same as using them. The simplest test, needle in a haystack, hides one fact at a chosen depth in long irrelevant text and asks for it. Plotting accuracy by length and depth exposes blind spots; models often miss facts in the middle more than at the start or end.

Passing is necessary but far from sufficient: retrieving one distinctive sentence is much easier than combining scattered facts, following chains of references, or tracking a quantity through a document. Harder benchmarks such as RULER test these and show many models' effective context well short of the advertised length. Sections 15.2 and 16.3 cover the architecture behind long contexts.

13.5 Judging Open-Ended Answers

Many tasks, such as writing an email or summarizing a report, have no single correct answer.

Human preference. People pick the better of two unlabelled answers, and the Bradley–Terry model of section 3.6 turns many such votes into ratings; Chatbot Arena ranks public models this way from millions of crowd votes.
LLM-as-judge. A strong model scores answers against a rubric. It is cheap and agrees with humans reasonably often, but it favors the answer shown first, longer answers, and its own style. Mitigations: evaluate both orderings, control for length, and check against a sample of human judgments.

Both measure preference, not correctness: a fluent, confident, wrong answer can win.

13.6 How Sure Is a Score?Run this section

Less sure than a single number suggests. A benchmark of $n$ questions samples the skill it measures. If each question is answered correctly with probability $p$ , each is a 0-or-1 variable with variance $p(1-p)$ , and by section 3.2 the accuracy has standard error

\text{SE} = \sqrt{\frac{p(1 - p)}{n}}

introduces 1 notation

An approximate 95% confidence interval is the accuracy plus or minus two standard errors. At 70% on 200 questions, , so the interval is roughly 64% to 76%, and models scoring 70% and 73% cannot be told apart.

Figure 13.6

A benchmark score is a sample

A model with a fixed true accuracy takes 2,000 fresh benchmarks of the chosen size. The histogram shows the scores it gets; the band is the predicted 95% interval.

Each question is answered correctly with probability equal to the true accuracy, independently.

Predict, then try

A model's true accuracy is 70%. On a 100-question benchmark, guess how far the measured score typically wanders from run to run of fresh questions, then try 1,000 questions.

In words. Fresh benchmarks of a chosen size are drawn for a model of chosen true accuracy and their scores histogrammed, against a shaded band for the predicted 95% interval.

Sampling randomness in generation (run several samples or use greedy decoding, and say which) and prompt sensitivity (section 13.2) add more noise. A careful report gives the interval, the number of samples and the prompt.

Exercises

Check

Why can't the perplexity of a model with a 32,000-token vocabulary be compared with that of a model with a 256,000-token vocabulary on the same text?

Apply

What is the approximate 95% confidence interval half-width (two standard errors) for an accuracy of 70% measured on 200 questions?

Apply

Ten programs are sampled and 3 pass. Estimate pass@1 and pass@5.

Apply

A model's total negative log-likelihood on a 1,000-byte text is 1,100 bits. What is its bits per byte?

Extend

An LLM judge prefers answer A over answer B 65% of the time when A is shown first, and prefers B over A 55% of the time when B is shown first. What do you conclude, and how would you report the comparison?

Core Academic References

Hendrycks et al. (2021): Measuring Massive Multitask Language Understanding (MMLU). ICLR 2021. arXiv:2009.03300
Chen et al. (2021): Evaluating Large Language Models Trained on Code (HumanEval, pass@k). arXiv:2107.03374
Cobbe et al. (2021): Training Verifiers to Solve Math Word Problems (GSM8K). arXiv:2110.14168
Liu et al. (2024): Lost in the Middle: How Language Models Use Long Contexts. TACL 12. arXiv:2307.03172
Hsieh et al. (2024): RULER: What's the Real Context Size of Your Long-Context Language Models?. arXiv:2404.06654
Zheng et al. (2023): Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. arXiv:2306.05685
Sainz et al. (2023): NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. Findings of EMNLP 2023. arXiv:2310.18018
Miller (2024): Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640