The block of Module 7 can be wired into three kinds of model. An encoder lets every token see the whole input, which suits understanding text. A decoder lets each token see only what came before, which suits writing it. The 2017 Transformer used both, joined by cross-attention. This module covers the three families, why decoder-only won for language models, and a small wiring choice, where the normalization goes, that decides whether deep models train at all.

8.1 The 2017 Encoder-Decoder and Cross-Attention

The 2017 Transformer was built for translation: an encoder reads the source sentence and a decoder writes the translation one token at a time. Each is a stack of six identical layers:

Architecture Flowchart

In words. Six encoder layers represent the source, each token seeing the whole sentence. In each of six decoder layers, a target token first attends to the target tokens before it, then across to the encoder's output. Each "Add & LayerNorm" adds a sub-layer's input to its output and normalizes.

Cross-Attention: Shapes and Mechanics

Cross-attention is how the decoder reads the source. Queries come from the decoder's vectors ( $N$ target tokens), keys and values from the encoder's output $H_{\text{enc}} \in \mathbb{R}^{M \times d_{\text{model}}}$ ( $M$ source tokens):

Q_{\text{cross}} = H_{\text{dec}} W_{\text{cross}}^Q \in \mathbb{R}^{N \times d_k} \quad (\text{Queries come from the Decoder})

introduces 1 notation

K_{\text{cross}} = H_{\text{enc}} W_{\text{cross}}^K \in \mathbb{R}^{M \times d_k} \quad (\text{Keys come from the Encoder})

V_{\text{cross}} = H_{\text{enc}} W_{\text{cross}}^V \in \mathbb{R}^{M \times d_v} \quad (\text{Values come from the Encoder})

The weights form an $N \times M$ table, one row per target token and one column per source token, each entry between 0 and 1:

A_{\text{cross}} = \operatorname{Softmax}\left(\frac{Q_{\text{cross}} K_{\text{cross}}^\top}{\sqrt{d_k}}\right) \in [0, 1]^{N \times M}

introduces 4 notations

There is no causal mask here. The decoder's self-attention must hide future target tokens, but the source is complete before decoding starts, so every target token may attend to every source token.

8.2 Encoder-Only: BERT and Masked Language ModelingRun this section

BERT (Devlin et al., 2019) keeps only the encoder, with no causal mask, and uses a different self-supervised task: hide about 15% of the tokens and predict them from everything else. With $\mathcal{S}$ the hidden positions and $W_{\setminus \mathcal{S}}$ the text with them hidden:

\mathcal{L}_{\text{MLM}} = -\sum_{i \in \mathcal{S}} \log P_\theta(w_i \mid W_{\setminus \mathcal{S}})

introduces 6 notations

Of the chosen tokens, 80% become a special [MASK] token, 10% a random token and 10% stay unchanged, so the model cannot learn that only [MASK] positions matter. Seeing both sides makes encoders strong at classification, search, tagging and embeddings, but they cannot generate text left to right, since nothing trains them to predict from the past alone. Encoders reappear wherever a model only has to read: image encoders in Module 25 and text recognizers in Module 27.

8.3 Encoder-Decoder After 2017: T5

T5 (Raffel et al., 2020) treats every task as text in, text out: translation, summarization, classification (the label written as a word) and question answering share one model and one loss.

Its pretraining task, span corruption, sits between BERT's blanks and next-token prediction: random spans are each replaced by one sentinel token, and the decoder writes the missing spans in order:

input: Thank you <X> me to your party <Y> week.
target: <X> for inviting <Y> last <Z>

Encoder-decoders remain strong when input and output are different kinds of sequence and the whole input is available up front: translation, speech recognition (Whisper is one) and summarization. But they spend parameters on both halves, and decoder-only models proved simpler to scale and just as capable once large, which is why this course's language models are decoder-only.

family	attention inside the model	typical pretraining task	good at
encoder-only (BERT)	every token sees every token	fill in masked tokens	classification, search, embeddings
encoder-decoder (T5)	encoder: all; decoder: causal + cross	fill in masked spans	translation, summarization, speech
decoder-only (GPT)	causal	predict the next token	generation, chat, general use

8.4 Decoder-Only: Predicting the Next TokenRun this section

The GPT models keep only the decoder: a stack of masked self-attention blocks trained to predict the next token. GPT-1 (2018), GPT-2 (2019) and GPT-3 (2020) showed that this design keeps improving with size and data, and most large language models since, including Meta's LLaMA family, use it.

By the chain rule of probability (section 3.3), the probability of a text of $T$ tokens $W = (w_1, \dots, w_T)$ is a product of next-token probabilities:

P(w_1, w_2, \dots, w_T) = \prod_{t=1}^T P(w_t \mid w_1, w_2, \dots, w_{t-1})

introduces 2 notations

$\prod$ means "multiply these together". So a model that answers "what is the next token, given everything so far?" can score any text and generate text of any length. Training maximizes the likelihood of real text (section 3.4) by minimizing the negative log-likelihood (NLL):

\mathcal{L}_{\text{NLL}}(\theta) = -\sum_{t=1}^T \log P_\theta(w_t \mid w_{<t})

$w_{<t}$ means all tokens before position $t$ . This is the cross-entropy loss of section 3.5 applied at every position, with the gradient of section 4.3, and is the perplexity.

Architecture Flowchart

In words. The whole sequence goes in at once, the mask hides each position's future, and the loss is computed at every position in one pass. Training is parallel even though generation is sequential.

The Causal Attention Mask Matrix

In training the model sees all $N$ tokens at once, but position $i$ predicts token $i+1$ and must not see it, or it could copy the answer. The causal mask of section 6.4 prevents this (row $i$ is the looking token, column $j$ the token looked at):

M_{ij} = \begin{cases} 0, & \text{if } j \le i \\ -\infty, & \text{if } j > i \end{cases}

introduces 3 notations

Added inside the softmax, as in Module 6:

\operatorname{MaskedAttention}(Q, K, V) = \operatorname{Softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right) V

introduces 1 notation

every future position gets exactly zero weight:

e^{-\infty} = 0 \implies \operatorname{Softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)_{ij} = 0 \quad \forall j > i

introduces 2 notations

( $\forall$ means "for all".) Each row's remaining weights still sum to 1. So one parallel pass gives every position the view it would have during generation, and a 1,024-token training sequence yields 1,024 predictions and loss terms from a single forward pass.

Figure 8.4

Three masks: who may look at whom

Rows are queries, columns are keys. Shaded cells are allowed (0 in the mask); blank cells are blocked (−∞). Hover or tap a row.

For cross-attention the rows are target tokens and the columns are the encoder’s source tokens; there is no causal mask, because the source is complete before decoding starts.

Predict, then try

Switch to the causal mask and hover the third row. Before you do, say how many keys it can see, and how many next-token predictions one 6-token training sequence gives.

In words. An attention mask as a grid, queries as rows and keys as columns, in three settings: bidirectional (encoder), causal (decoder) and cross-attention. Hovering a row highlights the keys it can see and, for the causal mask, names the token that position predicts.

8.5 Where the Norm Goes: Post-LN and Pre-LNRun this section

LayerNorm can go in one of two places around each residual addition. The 2017 Transformer put it after, Post-LN:

\mathbf{x}_{l+1} = \operatorname{LayerNorm}(\mathbf{x}_l + \operatorname{SubLayer}(\mathbf{x}_l))

introduces 2 notations

$\operatorname{SubLayer}$ is attention or the feed-forward network, and $l$ counts sub-layers, not blocks. GPT-2 (2019) moved LayerNorm to each sub-layer's input, Pre-LN, as GPT-3 and LLaMA do (LLaMA with RMSNorm, section 15.3):

\mathbf{x}_{l+1} = \mathbf{x}_l + \operatorname{SubLayer}(\operatorname{LayerNorm}(\mathbf{x}_l))

Pre-LN never normalizes the main path, so it adds one final LayerNorm, ln_f, before the output layer. Xiong et al. (2020) explained why Pre-LN trains more easily: in Post-LN the gradients near the output are large at the start, so the learning rate must be ramped up slowly (a warmup); Pre-LN can often train without one.

In the reference implementation. src/transformer/transformer_block.py uses the Post-LN wiring from 2017, so on this point it differs from GPT-2. Lab 3b runs the two wirings against each other.

Architecture Flowchart

In words. Post-LN normalizes the sum, so the normalization sits on the main path. Pre-LN normalizes only the copy entering the sub-layer, leaving the main path a plain addition.

Mathematical Proof of Gradient Highway in Pre-LN

In Pre-LN nothing on the residual stream is normalized, so unrolling across $L$ sub-layers gives the input plus every layer's contribution:

\mathbf{x}_L = \mathbf{x}_0 + \sum_{l=0}^{L-1} \operatorname{SubLayer}(\operatorname{LayerNorm}(\mathbf{x}_l))

Differentiate with respect to $\mathbf{x}_0$ ; the first term gives the identity $I$ :

\frac{\partial \mathbf{x}_L}{\partial \mathbf{x}_0} = I + \sum_{l=0}^{L-1} \frac{\partial \operatorname{SubLayer}(\operatorname{LayerNorm}(\mathbf{x}_l))}{\partial \mathbf{x}_0}

introduces 2 notations

$I$ depends on no layer's weights, so part of the gradient always reaches the bottom unscaled. In the RNN of section 5.2 the gradient is a product of per-step factors; here the layer terms are added to $I$ , not multiplied into it. Post-LN has no such clean term, because the gradient passes back through a LayerNorm at every layer.

Caveat. The $I$ term protects only the residual stream, the direct route from $\mathbf{x}_0$ to $\mathbf{x}_L$ . The other terms in the sum each pass through a LayerNorm and a sub-layer, and those still rescale the gradient.

Measured (lab 3b, experiments/ab_layernorm.py). The same 49,792 parameters, wired two ways, give outputs that differ by $0.83$ at initialization. The identity path shows up as a gradual trend, not a sudden break. As depth grows from 4 to 32 layers, the gradient reaching the embeddings changes by a factor of $0.9749$ under Post-LN and grows by a factor of $1.6227$ under Pre-LN, so the ratio between the two widens from $1.5$ to $2.5$ .

Measured (lab 3b). At this size the effect is modest: at 32 layers Post-LN's gradient is 2.5 times smaller than Pre-LN's, not gone. A 600-step training comparison at two layers separates the two wirings by $0.015$ validation loss, which is within the run-to-run noise of a single seed.

Exercises

Check

Which family would you choose for (a) classifying support tickets into ten categories, (b) translating documents, (c) a chat assistant? Give one reason each.

Apply

Write out the causal mask for four tokens and say how many entries are $-\infty$ .

Apply

A decoder-only model is trained on one sequence of 2,048 tokens. How many next-token predictions contribute to the loss in that single forward pass?

Apply

Unroll the Pre-LN update $\mathbf{x}_{l+1} = \mathbf{x}_l + f_l(\operatorname{LN}(\mathbf{x}_l))$ over three sub-layers and write $\mathbf{x}_3$ in terms of $\mathbf{x}_0$ .

Extend

BERT replaces only 80% of its chosen tokens with [MASK], 10% with random tokens and leaves 10% unchanged. What would go wrong if it used [MASK] for all of them?

Core Academic References

Vaswani et al. (2017): Attention Is All You Need. NeurIPS 2017. arXiv:1706.03762
Devlin et al. (2019): BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. arXiv:1810.04805
Raffel et al. (2020): Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5). JMLR 21(140): 1–67. arXiv:1910.10683
Radford et al., OpenAI (2018): Improving Language Understanding by Generative Pre-Training (GPT-1). Technical Report.
Radford et al., OpenAI (2019): Language Models are Unsupervised Multitask Learners (GPT-2). Technical Report.
Radford et al., OpenAI (2023): Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). ICML 2023. arXiv:2212.04356
Xiong et al. (2020): On Layer Normalization in the Transformer Architecture (Pre-LN analysis). ICML 2020. arXiv:2002.04745

Three Families: Encoder, Encoder-Decoder, Decoder

8.1 The 2017 Encoder-Decoder and Cross-Attention

Cross-Attention: Shapes and Mechanics

8.2 Encoder-Only: BERT and Masked Language ModelingRun this section

8.3 Encoder-Decoder After 2017: T5

8.4 Decoder-Only: Predicting the Next TokenRun this section

The Causal Attention Mask Matrix

8.5 Where the Norm Goes: Post-LN and Pre-LNRun this section

Mathematical Proof of Gradient Highway in Pre-LN

Exercises

Core Academic References

Measured labs

Lab 3b — where LayerNorm goes (Post-LN vs Pre-LN), up to 32 layers