Reading a Model Card

An open model's card and configuration file answer most of what this course has taught you to ask: how wide, how deep, which attention, which position encoding, how big a vocabulary. This module compares the families you have met, then reads a configuration field by field and turns it into parameters, KV cache and compute per token.

19.1 Comparing the Architectures

The table collects the choices of Modules 15 to 17 for four families.

Every row is from the cited papers and model cards, not measured; none of these architectures is in the reference repository.

In words. The explorer lists architectures from the original Transformer and GPT-2 to LLaMA 3, Gemma 2, Qwen, a vision-language model, PP-OCR and Qwen3-TTS, and shows the chosen one's normalization, position encoding, activation, attention, vocabulary and loss beside the others.

19.2 A Config File, Field by FieldRun this section

Models on Hugging Face ship a config.json. Here is one for an 8-billion-parameter LLaMA-3-style model:

{
  "vocab_size": 128256,
  "hidden_size": 4096,
  "num_hidden_layers": 32,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "intermediate_size": 14336,
  "hidden_act": "silu",
  "max_position_embeddings": 8192,
  "rope_theta": 500000.0,
  "rms_norm_eps": 1e-05,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16"
}

Other fields signal Modules 16 to 18: sliding_window or per-layer attention types for local and global layers, num_experts and num_experts_per_tok for mixture of experts, rope_scaling for context extension, and final_logit_softcapping for Gemma's soft cap.

19.3 Estimating Parameters, Cache, and ComputeRun this section

With $d =$ hidden_size, $L =$ num_hidden_layers, $d_h = d / h_q$ and $d_{ff} =$ intermediate_size:

• Parameters per block. Attention has $W^Q$ and $W^O$ of size $d \times d$, and $W^K$ and $W^V$ of size $d \times h_{kv} d_h$, so $2d^2 + 2 d\,h_{kv} d_h$. A SwiGLU feed-forward network has three matrices, $3\,d\,d_{ff}$. Norms add a few times $d$.
• Total parameters. $L$ blocks plus the embedding table $\lvert V\rvert d$, plus the same again for the output layer if it is not tied.
• KV cache per sequence (sections 15.5 and 23.1): $2 \times L \times h_{kv} \times d_h \times (\text{context}) \times (\text{bytes})$.
• Compute per token, from section 10.5: about $2 \times$ (parameters) FLOPs in the forward pass, plus the attention scores, about $2 \times L \times d \times (\text{context})$ more, which matters only for long contexts.

Worked example. For the configuration above: attention is $2 \cdot 4096^2 + 2 \cdot 4096 \cdot 8 \cdot 128 = 41.9$ million per block, SwiGLU is $3 \cdot 4096 \cdot 14{,}336 = 176.2$ million, so a block is about 218 million and 32 blocks about 6.98 billion. The untied embedding and output layer add $2 \cdot 128{,}256 \cdot 4096 = 1.05$ billion, for a total of about 8.03 billion: the "8B" in the name. At 8,192 tokens in bf16 the KV cache is $2 \cdot 32 \cdot 8 \cdot 128 \cdot 8192 \cdot 2$ bytes $= 1$ GiB per sequence.

In words. An editable model configuration, with presets, and a readout of head width, parameters per block and in total, KV cache at a chosen context, and forward compute per token.

Exercises

Core Academic References

• Radford et al., OpenAI (2019): Language Models are Unsupervised Multitask Learners (GPT-2). Technical Report.
• Grattafiori et al. (2024): The Llama 3 Herd of Models. arXiv:2407.21783
• Gemma Team, Google DeepMind (2024): Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118
• Qwen Team, Alibaba (2024): Qwen2.5 Technical Report. arXiv:2412.15115
• Mitchell et al. (2019): Model Cards for Model Reporting. FAT* 2019. arXiv:1810.03993