CLIP

The objective#

For a batch of N pairs, with L2-normalised embeddings z_i for images and z_t for texts and a learned temperature tau:

logits[i][j] = ( z_i[i] . z_t[j] ) / tau

L_i2t = -(1/N) * sum_i  log(  exp(logits[i][i]) / sum_j exp(logits[i][j])  )
L_t2i = -(1/N) * sum_j  log(  exp(logits[j][j]) / sum_i exp(logits[i][j])  )

L = (L_i2t + L_t2i) / 2

z_i[i] is the projected, normalised image embedding of pair i. z_t[j] is the projected, normalised text embedding of pair j. tau is a scalar temperature, parameterised in CLIP as exp(t) with t learned and clamped to keep 1/tau below 100. N is the batch size. The diagonal entries are the positives; everything else is treated as a negative.

This is InfoNCE (van den Oord et al., 2018) applied symmetrically across two views. Minimising it maximises a lower bound on the mutual information between the two modalities, and the bound is capped at log N — which is the formal reason batch size matters so much.

What CLIP actually did#

Radford et al. (2021) trained on WIT, 400 million image-text pairs collected from the public web, using 500,000 queries and capping each query at 20,000 pairs to balance the distribution. Neither the dataset nor the collection code was released.

Architectures: a ResNet-50 family and a ViT family for images, and a 63M-parameter Transformer for text, with the end-of-sequence token's final layer taken as the text embedding. Both towers project to a shared 512-dimensional space (768 for the largest ViT). Batch size was 32,768, sharded across GPUs with an all-gather so every device sees the full negative set.

The headline result: ViT-L/14 at 336 pixels matches a fully supervised ResNet-50 on ImageNet without seeing a single ImageNet training label. The more important result is the robustness gap — on distribution shifts such as ImageNet-A, ImageNet-R and ObjectNet, CLIP degrades far less than supervised models with equal in-distribution accuracy.

Prompt engineering as a measurable effect#

The paper reports that switching from a bare class name to the template "a photo of a {label}." gains roughly 1.3 points of ImageNet accuracy, and ensembling 80 templates by averaging their normalised text embeddings gains about 3.5 points in total. This is a training-free change that costs one extra forward pass per class.

The mechanism is a distribution-shift correction. Web captions are sentences; a bare noun sits outside the text encoder's training distribution.

The temperature, in detail#

A logit scale of 1/tau = 100 is CLIP's clamp. Larger scales sharpen the softmax and concentrate gradient on the hardest negative; smaller scales spread it. Wang and Liu (2021) show the temperature controls a hardness-aware property of the loss — low temperature makes it approach a hard-negative-mining objective, which improves discrimination but hurts tolerance to semantically similar negatives.

That trade-off has a practical face. A very low temperature pushes apart two near-duplicate captions in the batch that genuinely describe the same image. These false negatives are not modelled by the loss at all, and their frequency rises with batch size.

Known limitations, measured#

• Compositionality. ARO (Yuksekgonul et al., 2022) and Winoground (Thrush et al., 2022) show near-chance or below-chance performance on relation and order tasks.
• Counting. Accuracy drops sharply beyond three or four instances. A pooled global embedding has no counting mechanism.
• OCR and fine text. Base variants read large text and fail on small text; this drove the resolution increases in later models.
• Bias. The paper's own Section 7 reports demographic disparities in zero-shot classification and cautions against surveillance use. Birhane et al. (2021), auditing LAION-400M, documented explicit and harmful content surviving automated filtering.

What came after#

ALIGN (Jia et al., 2021) used 1.8 billion noisier pairs with no curation and matched or beat CLIP, arguing scale substitutes for filtering. OpenCLIP (Ilharco et al., 2021) reproduced the recipe on public LAION data and published the first honest scaling curves for this family. SigLIP (Zhai et al., 2023) replaced the batch softmax with a pairwise sigmoid loss, removing the all-gather and making small-batch training viable. EVA-CLIP and DFN pushed data filtering further, consistent with DataComp's finding that data selection dominates architecture at fixed compute.

Cost#

Original CLIP training: 592 V100 GPUs for 18 days for the largest ViT model, and 256 V100s for 12 days for the largest ResNet. Inference is cheap by comparison — one ViT-B/32 forward pass is roughly 4.4 GFLOPs per image, which is why the gallery side is precomputed and only the query is encoded live.

Papers#

• Radford et al., Learning Transferable Visual Models From Natural Language Supervision, 2021 — arxiv.org/abs/2103.00020
• van den Oord, Li and Vinyals, Representation Learning with Contrastive Predictive Coding, 2018 — arxiv.org/abs/1807.03748
• Jia et al., Scaling Up Visual and Vision-Language Representation Learning with Noisy Text Supervision (ALIGN), 2021 — arxiv.org/abs/2102.05918
• Wang and Liu, Understanding the Behaviour of Contrastive Loss, 2021 — arxiv.org/abs/2012.09740
• Birhane, Prabhu and Kahembwe, Multimodal Datasets: Misogyny, Pornography, and Malignant Stereotypes, 2021 — arxiv.org/abs/2110.01963
• Zhai et al., Sigmoid Loss for Language Image Pre-Training, 2023 — arxiv.org/abs/2303.15343

What to learn next#

• Vision-language models — bolting a CLIP-style encoder onto a language model.
• Cross-modal retrieval — putting a shared space to work at scale.
• Text to image — where the text encoder ends up next.