II · THE IDEA · ARTIFICIAL INTELLIGENCE
Vision Transformer (ViT) Scaling
▶ Listen · narrated
Convolutional networks had long seemed the natural architecture for images. Feeding a transformer nothing but patch embeddings challenged that assumption — and revealed how much the result depends on scale.
At a glance
- What it is
- A standard transformer encoder run on linearly embedded image patches
- Patch size
- Often 16×16 pixels; each patch becomes one token
- Inductive bias
- Almost none beyond the patch cut and position embeddings
- Scale finding
- Large pre-training data closes and then reverses the CNN advantage
- Scaling paper
- Maps accuracy against model size, data size and compute
Think of a photograph cut into a chessboard of small squares. Each square is squashed into a list of numbers, and those lists are handed to a transformer — the same kind of model used for sentences — as if each square were a word. The model never runs a convolution filter across neighbouring pixels. It only looks at relationships among the patch-tokens, using attention.
That is a Vision Transformer. Because it is not told in advance that nearby pixels matter more than distant ones, it has to learn spatial structure from examples. When the training set is only moderately large, convolutional networks, which are built around local filters, often still win. When pre-training uses far more images, the transformer catches up and can surpass them on transfer tasks, because its flexibility starts to outweigh the missing prior. A later study charted this carefully: grow the model, grow the data, and measure how accuracy moves. The lesson is joint scale, not architecture slogans alone.
ViT tokenises an image x ∈ ℝ^{H×W×C} into N non-overlapping patches of resolution P×P, flattens each patch, and applies a learned linear projection E to obtain patch embeddings of dimension D. A positional embedding E_pos is added; optionally a class token x_class is prepended. The sequence z_0 = [x_class; x_p¹E; …; x_pᴺE] + E_pos is processed by L standard pre-norm (or post-norm) transformer encoder blocks — multi-head self-attention and MLP, with residual connections. Classification uses the final class token (or mean-pooled patch tokens) through an MLP head.
Sequence length is N = HW/P², so attention costs scale as O(N²). Smaller P improves spatial resolution at quadratic cost. There is no convolutional stem in the pure variant; hybrids exist but are a separate design point. On ImageNet-scale pre-training without strong regularisation, ResNets of similar compute often outperform ViT. When pre-trained on much larger datasets (e.g. JFT-300M in the original study) and transferred, ViT matches or exceeds CNN baselines at favourable training cost in the reported comparisons.
The scaling paper sweeps model size, dataset size and total compute, showing that larger ViTs benefit disproportionately from more data, that under-trained large models are inefficient, and that transfer representation quality continues to improve over the studied range. Practical knobs remain patch size, position-embedding scheme, whether to use a class token versus pooling, and upstream data volume. Limitations inherited from the architecture include quadratic attention in the number of patches, weaker sample efficiency at small scale, and dependence on learned positional structure for any notion of grid geometry.
Look closer
The image becomes a short sequence
An input image is cut into a fixed grid of patches — 16×16 pixels is a common choice. Each patch is flattened and passed through a learned linear projection to produce one embedding vector, exactly as a token embedding would. A learnable class token can be prepended, and position embeddings are added so the encoder knows where each patch sat in the grid. From that point the architecture is an ordinary transformer encoder: multi-head self-attention and MLP blocks, with no convolutions anywhere in the stack.
The missing convolutional prior
Convolutional networks bake in locality and translation equivariance. A Vision Transformer does not. Early layers must learn spatial structure from data rather than inheriting it from the architecture. On mid-sized datasets this is a handicap: with comparable compute, carefully tuned ResNets often win. The original study is explicit that the pure transformer only pulls ahead once pre-training moves to much larger corpora, where the flexible architecture can absorb patterns the CNN's fixed prior would have constrained.
How the scaling study reads the trade-offs
The follow-up scaling work trains Vision Transformers across a wide range of model sizes and dataset sizes and plots transfer accuracy against those axes and against total training compute. Larger models improve more when given more data; under-fed large models waste capacity. Representation quality, measured by transfer to held-out tasks, keeps rising with scale in the regimes they study, which is the practical warrant for pushing further rather than stopping at ImageNet-sized pre-training.
The story
For most of the 2010s, strong image recognition systems were convolutional. Local filters, weight sharing and a hierarchy of expanding receptive fields matched what practitioners believed about natural images, and the empirical record agreed. The Vision Transformer paper asked a blunt counterfactual: what happens if those inductive biases are stripped out and a standard transformer encoder is pointed at images instead?
The method is almost austere. An image is divided into non-overlapping patches of fixed resolution. Each patch is flattened into a vector and multiplied by a learned matrix to yield a single embedding of the model's hidden size. Position embeddings — learned vectors indexed by patch location — are added so that spatial arrangement is not discarded. The resulting sequence is fed to a transformer encoder identical in structure to those used for language. A special class token, or global pooling over patch tokens, supplies the representation used for classification.
Because self-attention mixes information globally from the first layer, the model can in principle relate any patch to any other immediately. That flexibility is double-edged. On datasets the size of ImageNet, without heavy regularisation, Vision Transformers tended to lag behind ResNets trained under similar budgets. The architecture has to discover locality, edge structure and translation patterns that a convolution is handed for free. The original paper reports that the balance tips when pre-training moves to much larger datasets: with enough data, the same pure transformer reaches or exceeds the transfer performance of convolutional networks while using less compute to train than comparable CNNs in the regimes they measured.
A year later, the scaling study treated that observation as a programme. It trained families of Vision Transformers at increasing depth and width, on datasets of increasing size, and recorded how upstream and transfer accuracy moved with model parameters, seen images and total training compute. The curves are not a single magic threshold; they are a set of relationships. Big models underperform small ones when starved of data. Feed them enough, and the larger capacity keeps paying off. Transfer representation quality continues to improve across the scales examined, which is why simply declaring a mid-sized ViT “done” after ImageNet pre-training understates what the architecture can do.
Two design details keep reappearing in both papers. Patch size sets the effective sequence length: finer patches give the model more spatial resolution and a longer sequence, at quadratic cost in attention. Position embeddings are the only built-in cue that the sequence was once a grid; remove them and performance collapses, which is one reminder that “no convolutional prior” is not the same as “no spatial information at all”. Everything else — the value of larger pre-training sets, the interaction of width and depth, the point at which extra parameters stop helping — is left for scale to decide.
The result is less a single model than a recipe: tokenise the image as a short sequence of patch embeddings, keep the encoder vanilla, and spend the saved architectural complexity on data and compute. Whether that bargain is worthwhile depends on the dataset you actually have. The papers do not claim transformers are universally better at vision. They claim that, once the data scale crosses a region their experiments map, the absence of a convolutional prior stops being a liability and becomes room to fit the statistics of the training distribution more closely.
Why it mattered then
When the first paper appeared, the dominant assumption in computer vision was that convolutional structure was not merely helpful but close to necessary for competitive accuracy at practical compute budgets. Hybrid models that sprinkled attention on top of CNN backbones were already in circulation; a pure transformer applied to raw patches was a cleaner and more provocative test. Demonstrating that large-scale pre-training could erase the CNN advantage, and do so with favourable training compute in the reported comparisons, reframed the architectural debate. The scaling follow-up then gave practitioners something rarer than a leaderboard win: measured curves relating parameters, data and compute to transfer accuracy, so that the next training run could be planned rather than guessed.
Why it matters now
Patch-token transformers are now a default backbone family for classification, detection, segmentation and multimodal systems that align images with language. The scaling relationships mapped in the second paper still shape decisions about when to grow the model versus when to gather more data, and about how fine a patch size is worth its sequence-length cost. Anyone choosing between a convolutional encoder and a ViT-style encoder on a finite dataset is replaying the same trade-off the original study measured: strong spatial priors help when data is scarce; flexible attention helps when data is not. The papers remain the clearest statement of that bargain.
The surprising detail
The architectural wager is almost inverted from language. In NLP, transformers succeeded with relatively modest inductive bias on corpora that were already huge. In vision, the same bias-light design looked weak until the data regime caught up — so the “victory” of ViT is less a proof that attention is inherently superior for images than a proof that scale can substitute for the priors convolutions provide. The scaling paper makes that substitution visible as curves, not slogans: capacity without data underperforms; data without capacity plateaus; the interesting operating region is the joint increase of both.
What is disputed
Exact crossover points where ViTs overtake CNNs depend on dataset, augmentation, regularisation and training recipe; the papers report results for particular setups and do not establish a universal data-volume threshold. Scaling curves are likewise empirical within the model and data ranges studied, not proofs about indefinite further growth.
Remember this
ViT turns an image into a sequence of patch tokens and relies on data scale to replace the spatial priors a CNN would have built in.
Test yourself
You have a specialised image dataset far smaller than the large pre-training corpora used in the ViT studies, and no practical way to obtain more labels. All else equal, why might a well-tuned convolutional network still be the more rational default, and what single change to your setup would most directly attack the reason?
On limited data the Vision Transformer's lack of convolutional locality and translation priors is a disadvantage: it must learn spatial structure that a CNN encodes by design, and the original study finds ViTs lagging ResNets in that regime. The most direct attack on that gap is not a cleverer optimiser but more pre-training data — or transfer from a ViT already pre-trained at large scale — so the flexible architecture can arrive with spatial structure already absorbed rather than having to discover it from the small labelled set alone.
Go deeper
- [2010.11929] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale · arxiv.org
- [2106.04560] Scaling Vision Transformers · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.