Vision Transformers and Contrastive Learning

This module shows how the same transformer reads images, and how image and text models are trained to understand each other: from convolutional networks to the Vision Transformer, which treats an image as a sequence of patches, and contrastive learning, which aligns image and text encoders using only pictures and their captions. Module 26 uses these encoders to give language models sight.

25.1 Images, Convolutions, and Inductive BiasRun this section

A colour image is an $H \times W \times C$ tensor (section 1.1), with $C = 3$ channels (red, green, blue) of brightnesses, often scaled to 0 to 1. A 224 by 224 image holds $150{,}528$ numbers. Fed as one long vector to an MLP, the first layer alone would need hundreds of millions of weights, and would have to learn separately that a cat's ear looks the same in every corner.

Convolutions

A convolution fixes both. It slides a small grid of weights, a filter or kernel (say 3 by 3), across the image, writing the dot product with each patch underneath into a feature map. A handful of weights, reused everywhere, detects the same pattern, such as a vertical edge, wherever it occurs. A layer has many filters, and the next layer's filters combine their maps: edges into textures, textures into parts, parts into objects.

In words. A small black-and-white image, an editable 3 by 3 filter (presets: vertical edge, horizontal edge, blur, sharpen) and the resulting feature map. Hovering an output cell highlights its pixel patch and shows the dot product.

A convolutional neural network (CNN) stacks convolutions, usually with pooling layers that shrink the maps. It builds in two assumptions: nearby pixels matter most, and a pattern means the same anywhere. Such built-in assumptions are inductive biases; they let CNNs learn from relatively little data, and CNNs dominated vision for a decade before transformers.

25.2 The Vision TransformerRun this section

The Vision Transformer (ViT) (Dosovitskiy et al., 2020) is the ordinary transformer of Module 7 with one new first step, which turns an image into a sequence.

In words. Cut the image into small square patches, flatten each, multiply by one shared matrix to the model's width, add a position vector, and run a normal transformer encoder. Each patch is one token.

The Patch Formulas

For an image $\mathbf{x} \in \mathbb{R}^{H \times W \times C}$:

1. Cut it into non-overlapping $P \times P$ patches, image area over patch area:

For $224 \times 224$ and $P = 16$, $N = 196$, a $14 \times 14$ grid.

1. Flatten each patch into $\mathbf{x}_p^{(i)} \in \mathbb{R}^{P^2 C}$: $16 \cdot 16 \cdot 3 = 768$ numbers.
2. Multiply each by the same learned matrix $E$ to width $d$, put a learned class token [CLS] in front, and add learned position vectors:

$E \in \mathbb{R}^{(P^2 C) \times d}$, and the semicolons stack rows, so $\mathbf{z}_0$ and $E_{pos} \in \mathbb{R}^{(N+1) \times d}$ have one row for the class token and one per patch. The encoder's final class-token vector summarizes the image, for classification, say.

In words. A small image cut into a four-by-four grid of 16-pixel patches. Each patch's red, green and blue planes of 256 numbers are laid end to end into 768 numbers, and multiplying by E makes one token. The encoder receives the class token, then one token per patch in reading order.

Multiplying by $E$ plays the role of section 1.5's embedding lookup. Beyond the patching, ViT has far fewer inductive biases than a CNN: its positions are learned and every patch attends to every other from the first layer, so it must learn from data which patches relate. The ViT paper found this pays off with a lot of data, while CNNs do better on smaller datasets.

Measured (lab 6, experiments/vit_patchify.py). The claim that ViT is "the standard transformer with almost no changes" can be tested, and it holds. Cutting a $224 \times 224$ RGB image into $P = 16$ patches gives a $14 \times 14$ grid of vectors, each $16^2 \times 3 = 768$ numbers long: $196$ tokens. Adding a [CLS] token makes $197$. Passing those $197$ vectors through src/transformer/transformer_block.py, with no changes to that file, gives an output of shape $(1, 197, 64)$: one image, $197$ tokens, each $64$ wide. The block cannot tell whether its input came from text or pixels. The only thing it requires of its input is the vector width, d_model.

25.3 Contrastive Learning: CLIP and SigLIPRun this section

Classification needs labelled images, which limits both the data's scale and what is learned (a fixed list of classes). CLIP (Radford et al., 2021) instead learns from the web's billions of captioned images, with no labelling.

It trains a ViT for images and a transformer for text, each producing one vector per input, projected to a shared width and normalized to length 1. For a batch of $N$ image-caption pairs, all $N \times N$ cosine similarities (section 1.4), scaled by a learned temperature $\tau$, form a matrix with the true pairs on the diagonal. Softmax cross-entropy over the captions for each image, and over the images for each caption, averaged, gives the InfoNCE (contrastive) loss:

where $s_{ij}$ is the similarity of image $i$ and caption $j$. Each true pair is pulled together and pushed from the other $N - 1$, which serve as free negative examples; bigger batches give more negatives and better models, and CLIP used 32,768.

SigLIP (Zhai et al., 2023) replaces the softmax with an independent sigmoid (section 3.6) on each entry, labelled 1 on the diagonal and 0 elsewhere. With no normalizing across entries, very large batches split across devices more easily, and SigLIP encoders are now common in vision-language models.

In words. Image and caption embeddings as two sets of arrows, beside their similarity matrix. Training on the contrastive loss rotates matching arrows together, brightens the diagonal and lowers the loss.

25.4 Zero-Shot ClassificationRun this section

A trained CLIP model can classify images into categories it never trained on: zero-shot classification.

1. Write one caption per candidate class, from a template: "a photo of a dog", "a photo of a cat", "a photo of a car".
2. Encode the captions with the text encoder and the image with the image encoder.
3. Predict the class whose caption is most similar to the image.

Classes are chosen at use time in plain language. On ImageNet, CLIP's zero-shot accuracy matched a ResNet-50 trained on its 1.28 million labelled images, without seeing any (Radford et al., 2021). Template wording matters, as prompt wording does (section 13.2), and averaging several templates helps.

More important here, contrastive training yields an image encoder whose vectors already line up with the meaning of words, the natural front end for a vision-language model (section 26.1).

Exercises

Core Academic References

• LeCun et al. (1998): Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11): 2278–2324.
• He et al. (2016): Deep Residual Learning for Image Recognition. CVPR 2016: 770–778.
• Dosovitskiy et al. (2020): An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT). ICLR 2021. arXiv:2010.11929
• Oord, Li & Vinyals (2018): Representation Learning with Contrastive Predictive Coding (InfoNCE). arXiv:1807.03748
• Radford et al. (2021): Learning Transferable Visual Models From Natural Language Supervision (CLIP). ICML 2021. arXiv:2103.00020
• Zhai et al. (2023): Sigmoid Loss for Language Image Pre-Training (SigLIP). ICCV 2023. arXiv:2303.15343