II · THE IDEA · ARTIFICIAL INTELLIGENCE
Multimodal Models
▶ Listen · narrated
When you attach an image to a prompt, the model does not look at it the way you do. It receives a long sequence of tokens derived from the picture, and those tokens are expensive.
At a glance
- What it is
- A separate neural network that converts image patches into vectors, then a projection layer that maps them into the language model's embedding space
- Patch size
- Commonly 16×16 pixels, following the Vision Transformer design
- Token cost
- A single image may produce hundreds of tokens, all occupying context window space
- Training approach
- Vision encoder often pretrained separately using contrastive learning between images and captions
Imagine you want to describe a photograph to someone who only speaks a language you do not know. You need a translator. In a multimodal model, the language model only understands tokens — the same kind it uses for text. It has no direct way to process pixels. So the system uses a separate component called a vision encoder to look at the image. The encoder chops the picture into small square patches, typically 16×16 pixels each, and converts each patch into a numerical representation called a vector. These vectors are then passed through a projection layer, which translates them into the language model's token space. After that translation, the language model can treat the image tokens just like text tokens, and it processes them all together in one sequence. The reason images are expensive is simple: a single image might turn into hundreds of tokens, and all of those tokens take up space in the context window.
A multimodal model that accepts images typically uses a three-stage pipeline. First, a vision encoder — often a Vision Transformer (ViT) — processes the image. The image is divided into fixed-size patches, commonly 16×16 pixels, which are flattened into vectors and passed through transformer encoder layers with learned positional embeddings. The output is a sequence of vectors, one per patch. For a 224×224 image with 16×16 patches, this produces 196 vectors in the vision encoder's embedding space.
Second, a projection layer maps these vision embeddings into the language model's token embedding space. This is typically a learned linear transformation or a small MLP, trained specifically to align the two embedding spaces. The projection layer is the only component that must be trained for the specific pairing of vision encoder and language model.
Third, the projected image tokens are concatenated with text tokens — usually with special tokens marking boundaries — and the combined sequence is passed to the language model. The model applies its standard attention mechanism over the entire sequence, treating image tokens and text tokens uniformly.
The vision encoder is usually pretrained separately using contrastive learning, as in CLIP. During multimodal training, the vision encoder and language model are often frozen, and only the projection layer and optionally some adapter parameters are trained. This modular approach allows reuse of existing pretrained components but means the system is not jointly optimised end-to-end. Token count scales with image resolution and patch size: halving the patch size quadruples the number of tokens. This makes high-resolution images expensive in terms of both context window budget and inference compute.
Look closer
The image is divided into fixed-size patches
The encoder does not process pixels one by one. It carves the image into a grid of square patches — typically 16×16 pixels each, though the size varies by architecture. Each patch is flattened into a vector and passed through a transformer encoder. A 224×224 image becomes 196 patches, so 196 tokens before any text has been added to the prompt. Larger images produce proportionally more tokens, and the context window fills quickly.
The vision encoder is trained separately, often with contrastive learning
The encoder is usually pretrained on image-caption pairs using a method called contrastive learning, where the model learns to align images with their descriptions. CLIP, described by Radford and colleagues, trained an image encoder and a text encoder jointly so that matching pairs scored high similarity and mismatched pairs scored low. The resulting vision encoder captures visual features that correlate with language, which is why it can later be connected to a language model through a projection layer.
The projection layer is a learned translation between two embedding spaces
The vision encoder outputs vectors in its own embedding space, which is not the same space the language model uses for text tokens. A projection layer — often a simple learned linear transformation, sometimes a small MLP — maps vision embeddings into the language model's token space. This layer is trained so the language model can treat image tokens as if they were text tokens, attending to them in the same way. The projection is specific to the pairing: a different language model would need a different projection layer.
The story
A language model processes sequences of tokens. It has learned rich representations of words, subwords and punctuation, but it has no mechanism for pixels. To make a multimodal model that can accept both text and images, the system needs a way to convert an image into tokens the language model can read.
The solution is a pipeline. First, a vision encoder — a separate neural network, often a Vision Transformer — processes the image. The image is divided into a grid of fixed-size patches, typically 16×16 pixels. Each patch is flattened into a vector and passed through transformer layers, much like text tokens in a language model. The Vision Transformer architecture, described by Dosovitskiy and colleagues, showed that this patch-based approach could match or exceed convolutional networks on image classification tasks, and it had the advantage of producing a sequence of vectors rather than a single pooled representation.
The vision encoder outputs one vector per patch. For a 224×224 image divided into 16×16 patches, that is 196 vectors. For a 512×512 image, over a thousand. Each vector captures information about its patch: edges, textures, colours, local structure. But these vectors live in the vision encoder's embedding space, not the language model's.
This is where the projection layer enters. It is a learned transformation, often a linear layer or a small multilayer perceptron, that maps each vision vector into the language model's token embedding space. After projection, the image tokens can be concatenated with text tokens and fed into the language model as a single sequence. The model attends to image tokens and text tokens in the same way, using the same attention mechanism it would use for an all-text prompt.
The vision encoder is typically pretrained separately. CLIP, described by Radford and colleagues, trained both an image encoder and a text encoder on hundreds of millions of image-caption pairs scraped from the internet. The training objective was contrastive: for each image, the correct caption should score higher similarity than any other caption in the batch, and vice versa. This forced the vision encoder to learn features that aligned with natural language descriptions. Once pretrained, the vision encoder can be frozen or fine-tuned when connected to a language model.
The cost of this architecture is context window space. A single image can consume hundreds of tokens. If your context window holds eight thousand tokens and an image costs five hundred, you have seventy-five hundred tokens left for conversation history, system instructions and the response. Attach multiple images and the budget shrinks further. This is not an implementation detail; it is a hard constraint that shapes what multimodal models can do in practice.
Why it mattered then
The architecture emerged from two lines of work that converged. Vision Transformers, published in 2020, showed that the transformer architecture — previously dominant in language tasks — could handle images by treating them as sequences of patches. CLIP, also from 2020, demonstrated that a vision encoder trained with contrastive learning on image-caption pairs could learn representations useful for a wide range of downstream tasks, without task-specific fine-tuning. Connecting the two was a natural step. The projection layer allowed a pretrained vision encoder to feed into a pretrained language model without retraining either from scratch. This modularity was appealing: vision encoders could improve independently, and different language models could be swapped in. It also meant that the enormous compute cost of training a large language model did not need to be repeated just to add vision capabilities.
Why it matters now
Multimodal models are now common in production systems. You can attach an image to a prompt in ChatGPT, Claude, Gemini and many open-weight models. The architecture described here — vision encoder, projection layer, language model — remains the standard approach, though implementations vary. Understanding it clarifies why images are expensive in terms of context, why image quality and resolution affect token count, and why a model's vision capabilities can improve without retraining the language model itself. It also explains a common failure mode: if the projection layer is undertrained or the vision encoder was pretrained on data that does not match the use case, the model may produce fluent text that does not accurately describe what is in the image. The language model is doing its job; the breakdown is in the translation from vision to language tokens.
The surprising detail
The vision encoder and the language model often never train together on the same objective. The vision encoder is pretrained with contrastive learning on image-caption pairs. The language model is pretrained on text. Only the projection layer — and sometimes a small number of additional parameters — is trained to connect them. This means the bulk of both models is frozen during multimodal training, which saves compute but also means the two components are not jointly optimised end-to-end. Some recent work has explored training vision and language components together from scratch, but the modular approach remains widespread because it allows reuse of existing pretrained models.
Remember this
Images become tokens through a separate encoder and a learned projection. Those tokens are expensive, and the context window feels it.
Test yourself
You send a high-resolution photograph and a low-resolution version of the same photograph to a multimodal model. The high-resolution image uses three times as many tokens. Name two distinct practical consequences beyond the obvious cost difference.
First, the high-resolution image consumes more of the context window, leaving less room for conversation history, instructions or a long response. If the window is eight thousand tokens and the image uses fifteen hundred instead of five hundred, you have lost a thousand tokens of capacity for everything else. Second, inference is slower, because the model must process more tokens in its forward pass. Attention scales quadratically with sequence length in standard transformers, so a longer sequence of image tokens increases compute per request. A third consequence: the high-resolution image may also provide more detailed information to the model, which could improve accuracy on tasks that depend on fine visual detail, but that benefit is not guaranteed and depends on whether the vision encoder and projection layer can usefully represent the additional detail.
Go deeper
- Learning Transferable Visual Models From Natural Language Supervision · arXiv · Alec Radford et al. · 2021-02-26
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale · arXiv · Alexey Dosovitskiy et al. · 2020-10-22
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.