α — a scale or weight whose meaning depends on the section
Read "alpha". The course reuses it, each time as in the original paper: the LoRA scale (1.3), the load-balancing weight (5.3), the SmoothQuant exponent (4.7), and the CTC forward table (7.2).
None of these meanings carries over to the next. In LoRA, scales the low-rank update. In mixture-of-experts training, (often 0.01) sets how much the balancing loss counts. In SmoothQuant, (usually 0.5) splits an outlier's size between activations and weights. In CTC, is a running total of path probabilities. The sentence around the symbol always says which one is meant.
Where the course uses it
- 5.6 The First Attention · Module 5, Sequences Before Transformersfirst use
- 10.5 Scaling Laws: How Big, and on How Much Text · Module 10, Pretraining
- 17.4 Router Collapse and the Load-Balancing Loss · Module 17, Mixture of Experts
- 20.3 LoRA and QLoRA · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
- 24.3 Outlier Channels and SmoothQuant · Module 24, Quantization and Number Formats
- 27.3 The CTC Objective and Its Recursion · Module 27, Recognition: OCR and Speech With CTC