The vision Transformer that needed 300 million images

A convolutional neural network, or CNN, is built to notice the same local pattern wherever it appears. The original Vision Transformer (ViT) removed that image-specific machinery. It cut an image into squares, called each square a token, and handed the resulting sequence to an almost ordinary language Transformer.

That simplicity is the paper's enduring idea. But the title's “at scale” is doing real work: the strongest model was pretrained on JFT, an internal dataset of 303 million images. On ImageNet-scale data, the authors found comparable CNNs ahead and larger ViTs sometimes worse.

For an image with height H, width W, and square patch width P, ViT creates HW/P² patches. Each patch is flattened and passed through the same learned linear projection. A learned classification token is prepended, and a position vector is added to every token so the model can recover where each square came from.

def patchify(image, patch_size):
    patches = []
    for top in range(0, len(image), patch_size):
        for left in range(0, len(image[0]), patch_size):
            patch = []
            for row in range(top, top + patch_size):
                for col in range(left, left + patch_size):
                    patch.append(image[row][col])
            patches.append(patch)
    return patches

The site runs the tested TypeScript implementation; the Python and C++ versions are line-for-line translations of its row-major patch extraction. The core also implements the affine patch projection, classification token, and position addition with hand-verified values.

A 224×224 image with 16×16 patches becomes 196 image tokens, plus one classification token. Smaller patches preserve finer detail, but attention compares every token with every token. Halving patch width makes four times as many image tokens and nearly sixteen times as many attention pairs.

Fine-tuning at higher resolution has the same effect. The paper keeps the patch size fixed, interpolates the learned position embeddings onto a larger grid, and accepts a longer sequence.

An inductive bias is a structural assumption that helps a model generalize before it has seen every possibility. CNNs bake in locality, two-dimensional neighborhoods, and translation equivariance—the idea that moving an object should move its features rather than change what the feature means. ViT mostly does not. Beyond cutting patches and interpolating positions, it must learn spatial relationships from examples.

The comparison is still meaningful: at matched JFT data, ViT-L/16 slightly exceeded BiT-L while using much less listed TPU compute. It is not a clean claim that Transformers always beat convolutions. Optimizer, regularization, resolution, model size, and access to pretraining data all move together.

• Hold compute fixed while varying patch size and dataset size.
• Measure robustness to one-pixel shifts across patch boundaries.
• Compare learned positions with explicit two-dimensional positions.

To inspect the attention mechanism that consumes these patch tokens, poke at the interactive transformer page. The previous FlashAttention walkthrough shows how the same token count becomes a systems bottleneck.