II · THE IDEA · ARTIFICIAL INTELLIGENCE
JEPA and I-JEPA
▶ Listen · narrated
Most self-supervised vision systems either fill in pixels or force two views of an image to agree. JEPA does neither: it asks a model to predict what is missing, but only in representation space.
At a glance
- What it is
- A JEPA applied to images: predict latent targets from context
- Not pixels
- Loss is computed in embedding space, not input space
- Three parts
- Context encoder, target encoder, and a predictor network
- Masking
- One context block; several target blocks sampled at multiple scales
- Target path
- Target encoder tracks the context encoder by exponential moving average
Think of a jigsaw in which you never look at the missing piece’s picture. You only try to guess a short description of what that piece must contain—sky, wheel, face—using the pieces still on the table.
I-JEPA does something like that with photographs. It hides parts of an image, reads the remaining part with one network (the context encoder), and reads each hidden part with another network (the target encoder). A third network tries to predict the target network’s description of each hidden part, using only the context and knowing where the gap is. Training scores the match between those descriptions, not a redraw of the actual pixels. Over time the context encoder gets good at forming descriptions that carry the kind of content needed to fill latent gaps—closer to objects and layout than to exact speckles of colour.
I-JEPA instantiates a joint-embedding predictive architecture for images. Given an image, a multi-block masking scheme produces one context region and several target blocks at multiple scales. The context encoder f_θ embeds the visible context. The target encoder f_θ̄ embeds each target block; θ̄ is an exponential moving average of θ, not a second set of freely trained weights under the same loss. A predictor g_φ, conditioned on the target’s positional encoding, maps the context embedding to a predicted target embedding. The training objective minimises a distance (as specified in the paper) between g_φ’s outputs and f_θ̄’s target embeddings. Gradients flow into θ and φ; θ̄ tracks θ through EMA.
Compared with masked autoencoders that decode to pixels or tokens, the loss never enters input space, so the optimisation target is alignment in representation space. Compared with contrastive or pure joint-embedding methods that enforce agreement across augmented views, I-JEPA’s supervisory signal is predictive and spatially localised: the same context representation must support several position-conditioned predictions. Limitations follow from the design: quality depends on the masking distribution and on EMA stability; the method assumes targets are predictable from context in latent space; and evaluation of “semantic” quality remains downstream and task-dependent rather than intrinsic.
Look closer
Prediction without generation
Generative self-supervision asks a model to rebuild missing pixels or tokens. I-JEPA instead maps both the visible context and the held-out targets into an embedding space, then trains a predictor to match the target embeddings from the context representation. Pixel-level texture never becomes the training objective, so the model is not rewarded for reconstructing exact colour or fine grain.
Context and targets are asymmetric
The context is a large block of the image with regions removed. The targets are smaller blocks sampled at several scales. The context encoder sees only the unmasked context. The target encoder sees the target blocks. The predictor receives the context embedding plus positional information for each target and must output a representation for that block. The two paths are not interchangeable.
Positional conditioning in the predictor
Because the predictor must say what belongs at a missing location, it is conditioned on where the target block sits in the image. The architecture therefore separates content learned by the encoders from location-specific prediction. What is learned is not a single global vector alone, but a capacity to fill in latent structure at specified positions.
The story
A Joint-Embedding Predictive Architecture, or JEPA, is a way of training encoders so that one part of a signal can be used to predict the representation of another part. The important word is representation. The system does not try to regenerate the missing signal in its original form. It tries to anticipate how an encoder would have embedded that missing piece.
I-JEPA is that idea applied to still images. An image is split, by masking, into a context region and one or more target regions. A context encoder processes the visible context. A target encoder processes each target block. A predictor network then takes the context embedding, together with information about the target’s position, and produces a predicted embedding for that block. Training minimises a distance between the predicted embedding and the target encoder’s actual output. Gradients update the context encoder and the predictor; the target encoder is not trained by the same path, but follows the context encoder through an exponential moving average.
That design choice matters. If both encoders were updated freely on the same loss, the system could collapse toward trivial constant representations that are easy to predict. Keeping the target path as a slowly moving average is one of the stabilising mechanisms that joint-embedding methods often need. I-JEPA also avoids the heavy hand-crafted view augmentations common in contrastive and pure joint-embedding work—random crops, colour jitter, and related transforms that define what counts as the same image. Instead, the multi-block masking itself supplies the predictive task: large semantic regions must be inferred from what remains visible.
Because the loss never asks for pixels back, the pressure on the model is different from masked autoencoding in the generative sense. Reconstructing pixels can be solved partly by modelling low-level statistics—local texture, edges, colour continuity. Predicting another encoder’s latent for a whole block pushes the representation toward whatever structure that encoder has learned to retain. In practice the paper presents this as a route to semantic features that transfer to recognition-style downstream work without requiring labels during pretraining.
The architecture is deliberately simple in outline: two encoders of the same family, a predictor conditioned on position, a masking scheme that samples context and targets at multiple scales, and a latent-space loss. The claim is not that generation is useless, but that for learning reusable image representations, prediction in embedding space is a more direct objective than rebuilding the input.
Why it mattered then
Self-supervised learning from images had largely settled into two families: generative methods that reconstruct missing or corrupted inputs, and joint-embedding methods that make two augmented views agree in representation space, often with a contrastive or clustering term. Both worked, and both carried costs. Generative objectives spent capacity on pixel detail that might not help recognition. View-agreement methods leaned on carefully designed augmentations and, in the contrastive case, on large batches or memory banks of negatives. I-JEPA offered a third pattern at a moment when those trade-offs were widely felt. By predicting in latent space from a masked context, it kept the predictive signal of masking without committing to pixel reconstruction, and it kept a joint-embedding structure without defining the task through hand-crafted dual views. The exponential-moving-average target encoder and the asymmetric context–target split were concrete engineering answers to collapse and to the need for a stable prediction target. In its own moment, the work mattered as a clear architectural proposal: semantics as the thing you predict, not the by-product of regenerating surfaces.
Why it matters now
Representation learning is still the bottleneck between raw sensory data and systems that reason, retrieve, or plan. Multimodal models continue to absorb vision backbones pretrained without labels, and the choice of pretraining objective still shapes what those backbones notice. A method that aims the loss at latent structure rather than at pixels remains relevant wherever compute is spent on features that will later be read by language models, detectors, or world models. JEPA-style prediction also travels beyond still images in principle: any setting where one can define context and targets in a shared embedding space can reuse the same outline. As interest grows in predictive world models and in encoders that support planning rather than only classification, the distinction between reconstructing observations and predicting representations stays practically useful. I-JEPA is one early, concrete demonstration of that distinction on images.
The surprising detail
The targets are not a single missing patch. I-JEPA samples several target blocks at different scales for each image, and the predictor must hit each of them from the same context representation, guided only by positional conditioning. The difficulty of the task is therefore tunable through geometry—how large the context is, how the targets are scattered—rather than only through the usual augmentation recipe. The “what is missing” signal is spatial and multi-scale by design.
What is disputed
The supplied sources establish the I-JEPA architecture and its stated motivation; they do not, by themselves, settle how it ranks against every later generative or contrastive variant on every benchmark. Transfer gains and the semantic character of the learned features should be read as claims of the paper, not as universal consensus.
Remember this
I-JEPA learns by predicting latent embeddings of masked image regions from context, not by reconstructing pixels.
Test yourself
A colleague suggests replacing I-JEPA’s latent loss with a pixel reconstruction loss on the same masked blocks, keeping the context encoder and masking unchanged. What objective pressure changes, and what failure mode of joint-embedding training might you no longer be managing in the same way?
The model is now rewarded for recovering local appearance—texture, colour, exact edges—rather than for matching another encoder’s representation of the block, so capacity can shift toward low-level detail that may not transfer as semantics. You also leave the joint-embedding setup: there is no longer a target encoder whose outputs must stay informative and non-collapsed under an EMA update; collapse of a second embedding space stops being the central risk, while generic, blurry, or texture-focused reconstructions become the new ones.
Go deeper
- [2212.07861] The Russian invasion of Ukraine selectively depolarized the Finnish NATO discussion · arxiv.org
- [2301.08243] Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.