[\mathbf{a}; \mathbf{b}]

[a; b] — concatenation

Read "a joined with b". A semicolon between items in square brackets means: join them end to end into one longer vector, or stack them as rows into one longer sequence.

$[\operatorname{head}_1; \dots; \operatorname{head}_h]$ places $h$ vectors, each $d_v$ wide, side by side into one vector $h\,d_v$ wide. That is why $W^O$ has $h\,d_v$ rows: one for each number in the joined vector.

When the items are matrices whose rows are tokens, the semicolon stacks their rows: $[\mathbf{X}_{\text{img}}; \mathbf{X}_{\text{text}}]$ is the image rows followed by the text rows, one sequence. The same notation adds a new row to the KV cache and inserts image patches among text tokens in a multimodal sequence. In PyTorch it is torch.cat.

Example

[[1, 2];\, [3, 4, 5]] = [1, 2, 3, 4, 5]

A length-2 and a length-3 vector join into a length-5 vector.

Where the course uses it

6.6 Multi-Head Attention · Module 6, Attentionfirst use
25.2 The Vision Transformer · Module 25, Vision Transformers and Contrastive Learning

Related terms

Query, key, valueThree learned versions of each token's vector. The query is what the token is looking for, the key is what it offers to others, and the value is what it passes on. W^Q is read "W Q", the matrix that makes queries.
The KV cacheRead "K cache" and "V cache". They store the keys and values of earlier tokens during generation, so each is computed once instead of again at every step.