Capstone: Build, Upgrade, Measure, Ship a Tiny GPT

This project puts the core track together in work you run yourself: train a small decoder-only model, upgrade it to the modern block one measured change at a time, then evaluate, adapt, shrink and speed it up, and look inside it. The deliverable is a short report in the course's style: every claim Measured, with its numbers and command, or marked Not measured, with a Caveat wherever a result should not be over-read.

The model is tiny, a character-level model of about 100,000 parameters trained on Shakespeare, so each experiment takes minutes on a free Colab CPU or a laptop. The companion notebook is a complete starting point: Module 7's 2017 block with every modern piece of Module 15 (Pre-LN, RoPE, RMSNorm, SwiGLU, grouped-query attention) as a switch, a training loop and an evaluation helper. The reference implementation, tiny-transformers, is the same design at a similar size.

Reread section 4.6 first. The most common failure here is not a bug but claiming a difference smaller than the run-to-run noise.

29.1 BuildRun this section

Goal: a trained baseline with an honest error bar.

1. Open the companion notebook and train the baseline configuration (character vocabulary, learned positions, Post-LN, GELU).
2. Record the validation loss and convert it to bits per character (section 13.1). Save the loss curves and check them against section 4.5: is anything spiking or flat?
3. Repeat the run with at least three seeds. Report the mean validation loss and its spread. This spread is the noise floor every later comparison must beat.
4. Count the parameters with the formula of section 7.6, and check it against the count the code reports.

Deliverable: a table of the three seeds, the mean and spread, and the parameter count by formula and by code.

29.2 UpgradeRun this section

Goal: move from the 2017 block to the 2024 block one change at a time, measuring each change as an ablation (section 4.6).

Apply the changes of section 15.6 in this order, each on top of the previous ones, with everything else held fixed (same data, steps, learning-rate schedule and seeds):

1. Post-LN → Pre-LN (section 8.5).
2. Learned positions → RoPE (section 15.1).
3. LayerNorm → RMSNorm (section 15.3).
4. GELU feed-forward → SwiGLU at matched parameter count (section 15.4).
5. Multi-head → grouped-query attention (section 15.5).

For each step, report the validation loss over three seeds, the parameter count, and whether the change is larger than the noise floor from 29.1. Then answer three questions in a paragraph each:

• Which changes made a measurable difference at this scale, and which did not? Section 4.6's four questions apply to your own boxes.
• RoPE's main benefit is behaviour at positions beyond the training length. Design and run a test of that (for example, evaluate at twice the training context) and report it.
• Which results would you expect to change at a thousand times the size, and why?

Deliverable: an ablation table, the length test, and the three paragraphs.

29.3 EvaluateRun this section

Goal: measure the upgraded model beyond its validation loss (Module 13).

1. Report bits per character on a held-out text from a different source than the training data. Explain why this is the right number to compare with any other model.
2. Build a small task suite of at least 50 prompts with checkable answers suited to a Shakespeare model: for example, completing well-known lines, or producing a valid character name after a speaker tag. Score it with exact match and report a 95% interval (section 13.6).
3. Check for contamination: are any of your prompts verbatim in the training text? Remove them, or report both scores.

Deliverable: both numbers with intervals, the task suite, and the contamination check.

29.4 AdaptRun this section

Goal: change the model's behaviour cheaply. Choose one.

• LoRA (section 20.3): fine-tune the upgraded model on a small text of a different style, training only rank-$r$ adapters on the attention projections. Compare $r = 1, 4, 16$ with full fine-tuning, in validation loss on the new text and in trainable parameters, and check how much the loss on the original text rises.
• DPO (section 21.6): define a simple preference (for example, completions that end at a line break are preferred over ones that stop mid-word), generate pairs from the model, and train with the DPO loss. Measure how often the preferred behaviour occurs before and after, and how far the model moved from the reference (the average log-ratio).

Deliverable: the comparison table and a short discussion of what the adaptation cost.

29.5 ShipRun this section

Goal: make the model smaller and faster, and measure what that costs.

1. Quantize the weights to INT8 and to INT4 with per-channel scales (section 24.2). Report the validation loss and the memory of each.
2. Implement generation with a KV cache (section 12.2), and test it the right way: compare logits position by position against the uncached model under teacher forcing, not the generated text.
3. Measure tokens per second with and without the cache at several sequence lengths, and explain the result using section 23.2 and the Measured box of section 12.2, which found only a small wall-clock gain at this size.

Deliverable: a table of size, loss and speed for each version, and the cache-correctness check.

29.6 ExplainRun this section

Goal: report one finding about what the model learned (Module 14).

1. Run the token tracer idea on your own trained model: pick a short input, extract every stage's tensor, and look at the attention patterns of every head.
2. Find and describe one interpretable behaviour. Candidates: a head that attends to the previous character, a head that attends to the start of the line, a head that looks back to the previous occurrence of the current character (an induction-like pattern on repeated words), or a logit-lens reading (section 14.4) showing at which layer the correct next character becomes the top guess.
3. Test the claim with an intervention: zero out the head (or its output) and measure what changes in the loss on inputs where the behaviour should matter, and on inputs where it should not.

Deliverable: the finding, the attention plots that show it, and the intervention result, written as a Measured box with a Caveat.

The Report

Assemble the six deliverables into one document, written the way this course is written:

• Every number has a source: a command, a seed list and a configuration.
• Every comparison states the noise floor it was measured against.
• Every result that might not hold at scale says so.
• Nothing you did not run is presented as if you had.

Compare your report with your answer to the "what do you expect?" questions of Module 0. The point of the course was to be able to write this report; the point of the report is to show that you can.

Exercises

Core Academic References

• Karpathy (2023): nanoGPT. GitHub repository.
• Bouthillier et al. (2021): Accounting for Variance in Machine Learning Benchmarks. MLSys 2021. arXiv:2103.03098
• Mitchell et al. (2019): Model Cards for Model Reporting. FAT* 2019. arXiv:1810.03993