CLIP's zero-shot classifier changes when you add one label

CLIP learns one space for pictures and language. An image encoder turns a picture into a vector; a text encoder does the same for a caption. If the vectors point in the same direction, their cosine similarityis high. Training rewards the real image-caption pairs and pushes apart every mismatched pairing in the same batch.

The clever part comes after training. To recognize dogs, planes, and birds, CLIP does not fit a new classifier on labeled images. It embeds text such as “a photo of a dog” and uses those text vectors as classifier weights. That is zero-shot classification: specifying a task in language without training on that task's labeled examples.

With N aligned image-text pairs, CLIP scores all N² combinations. The diagonal contains real pairs; the other entries are negatives. A row-wise cross entropy asks each image to find its caption, and a column-wise loss asks each caption to find its image. Their average is the symmetric CLIP loss.

def clip_loss(images, texts, scale):
    images = [normalize(vector) for vector in images]
    texts = [normalize(vector) for vector in texts]
    logits = [[scale * dot(image, text) for text in texts] for image in images]
    image_loss = diagonal_cross_entropy(logits)
    text_loss = diagonal_cross_entropy(transpose(logits))
    return (image_loss + text_loss) / 2.0

The site runs the tested TypeScript version; Python and C++ are faithful translations. For two perfectly aligned, orthogonal pairs at scale 2, the core returns 0.126928, exactly log(1 + exp(−2)).

Cosine similarity is bounded from −1 to 1, so CLIP learns a positive scale before the softmax. Multiplying every score widens the gaps: rankings stay fixed, but probabilities become sharper. This parameter is learned as a log-scaled number rather than hand-tuned.

A softmax probability is relative to every candidate. The toy below starts with an ambiguous “bat” embedding and two text classes: flying mammal and baseball field. The first wins decisively. Add a more specific baseball-bat description and it becomes the winner—even though the image vector and both original text vectors are unchanged.

Prompt wording matters for the same reason. The authors embed class names inside task-specific templates and ensemble several text embeddings per class. “Zero-shot” means no downstream image training, not zero design choices or zero validation feedback—the limitations section says the team repeatedly consulted validation sets during development.

The broader benchmark contains sharp failures. Relative to a supervised linear classifier on ResNet-50 features, zero-shot CLIP was 28.9 points better on Stanford Cars but 37.1 points worse on EuroSAT and 34.0 points worse on KITTI distance. Language is a flexible interface, not a guarantee that pre-training covered the requested task.

• Report sensitivity to label synonyms and prompt templates.
• Add plausible missing classes and audit which predictions flip.
• Measure each subgroup before and after changing the candidate set.

The text encoder is a Transformer, so you can inspect its mechanics on the transformer page. The final text-derived weights behave like a linear classifier; compare them with the logistic-regression page and follow the encoders on the neural-network page.