B

Batch — examples processed together

A group of examples processed at the same time. It adds one axis, of size B, at the front of every tensor. Every example in the batch goes through the same computation.

Hardware is fastest when it does many identical computations at once, so models process several sequences together. A batch of $B$ sentences, each $N$ tokens of width $d_{\text{model}}$ , is one tensor of shape $B \times N \times d_{\text{model}}$ .

The examples in a batch never mix, which is why most formulas in the course leave the batch axis out and describe one sequence. In training, each gradient step averages the loss over one batch (section 2.5).

Example

B = 8,\; N = 1024,\; d_{\text{model}} = 768 \;\Longrightarrow\; 8 \times 1024 \times 768

Eight sequences of 1,024 tokens, each token a vector of 768 numbers.

Where the course uses it

1.1 Scalars, Vectors, Matrices, and Tensors · Module 1, Shapes and Linear Algebrafirst use
2.4 Gradient Descent · Module 2, Calculus and Optimization
23.1 The Memory Wall · Module 23, Serving

Related terms

Tensor — numbers on any number of axesAn array of numbers with any number of axes. In code every array is called a tensor: a scalar has 0 axes, a vector 1, a matrix 2. In prose the word usually means 3 or more.
Shape and axesA tensor's shape lists the size of each axis, in order. (2, 3, 4) has three axes of sizes 2, 3 and 4. Checking shapes is the quickest way to follow a formula.
N, T, L — sequence lengthThe number of tokens in the sequence. Different modules call it N, T or L. N² ("N squared") is how attention's cost grows with it.