II · THE IDEA · ARTIFICIAL INTELLIGENCE
Stable Diffusion Architecture
▶ Listen · narrated
Pixel-space diffusion is powerful and expensive. Stable Diffusion’s lineage moves the generative process into a latent space so high-resolution images stay within practical compute.
At a glance
- Core idea
- Run diffusion in an autoencoder’s latent space, not on pixels
- Conditioning
- Cross-attention layers inject text (and other) signals into the UNet
- Two stages
- Train perceptual compression first, then the generative diffusion model
- Backbone
- Time-conditional UNet operating on latent tensors
- Payoff
- High-resolution synthesis at far lower cost than pixel-space diffusion
Think of writing a long story in rough shorthand first, then expanding the shorthand into polished prose at the end. Latent diffusion does something similar with images.
An autoencoder learns a compact stand-in for the picture—a latent—that keeps the important layout and content but is smaller to work with. A diffusion model then repeatedly cleans noise out of that compact version, steered by the text prompt through cross-attention layers inside a UNet. Only when the latent looks “clean” does a decoder turn it into the final grid of pixels.
So the model is not erasing noise from your megapixel image step by step. It is erasing noise from a smaller code, which is why high-resolution prompts became much cheaper to run than earlier pixel-space diffusion systems.
Latent diffusion factors image synthesis into a perceptual compression stage and a generative stage. An autoencoder provides an encoder E and decoder D such that z = E(x) is a lower-dimensional spatial latent and D(z) ≈ x. Diffusion is trained on z: noise is added according to a schedule, and a time-conditional UNet εθ(z_t, t, τ(y)) is optimised to predict the noise (or a related target), where τ(y) is a conditioning encoding of the prompt y.
Conditioning enters by cross-attention inside the UNet at multiple feature resolutions, so token-level text signals can modulate spatial activations rather than acting only as a global class embedding. Sampling draws z_T from a prior, applies the learned reverse process for T steps under fixed conditioning, and returns x̂ = D(z_0).
Limitations follow the split. Reconstruction error and latent capacity bound detail the diffusion model can never recover; aggressive compression hurts fidelity. The UNet still requires many sequential evaluations. Prompt alignment depends on the text encoder and on how well cross-attention was trained. None of this removes mode-covering or data-bias issues inherent to the training corpus; it relocates the expensive generative modelling into a space where each step is cheaper per output resolution.
Look closer
Compression is separated from generation
An autoencoder is trained first to map images into a lower-dimensional latent representation and back again. The diffusion model never sees raw pixels during sampling; it denoises latents, and only the decoder turns the final latent into an image. Mild compression keeps enough detail for fidelity while cutting the spatial work the UNet must do at every step.
Text arrives through cross-attention
Prompt tokens are encoded by a language model and supplied to the UNet via cross-attention at multiple resolutions. The same mechanism can accept other conditionings—layouts, classes—without redesigning the backbone. The denoiser therefore learns a joint space of noise level, spatial structure, and conditioning signal rather than treating text as a single global vector bolted on at the start.
The UNet still does the iterative work
Generation remains a multi-step denoising trajectory. At each step the UNet predicts noise (or a related target) in latent space, conditioned on timestep and text. Moving that loop into latents does not remove iteration; it shrinks what each iteration touches, which is why resolution and batch size become practical on more modest hardware than pixel-space diffusion required.
The story
Latent diffusion models attack a simple imbalance. Diffusion models learn to reverse a gradual noising process and can synthesise detailed images, but doing that directly in pixel space means every denoising step operates on a large grid. Cost scales with spatial resolution, and high-resolution sampling becomes heavy long before the model has said anything interesting about content.
The architecture associated with Stable Diffusion, introduced as latent diffusion models, splits the problem. A pretrained autoencoder first learns a perceptual compression: an encoder maps an image to a spatially smaller latent tensor, and a decoder reconstructs an image from that latent. Training this stage is about preserving what matters to human viewing while discarding high-frequency detail the generative model need not model explicitly. Once the autoencoder is fixed, a diffusion model is trained entirely in that latent space.
The generative backbone is a time-conditional UNet. Noisy latents enter; the network predicts the noise component (or an equivalent parameterisation) given the current timestep. Because the latent grid is smaller than the pixel grid, each forward pass is cheaper, and the same compute budget supports higher output resolutions or more sampling steps. After the reverse process finishes, the decoder lifts the clean latent back into image space in a single pass.
Text conditioning is not a separate head glued to the output. Token embeddings from a language encoder are fed into the UNet through cross-attention layers distributed across the network’s resolutions. That design lets the model bind words to spatial structure at several scales. The same cross-attention pathway can carry other conditionings, so the architecture is a general conditional generator rather than a text-only special case.
Training still follows the diffusion recipe: latents are noised according to a schedule, and the UNet is optimised to reverse that corruption under the chosen conditioning. At inference, one starts from pure noise in latent space, steps the learned reverse process while holding the text embedding fixed, then decodes. The expensive iterative work stays in the compressed domain; the expensive-looking image only appears at the end.
The paper frames this as a deliberate reallocation of capacity. Heavy generative modelling is reserved for the semantically rich latent, while a comparatively lighter autoencoder handles perceptual detail. That split is what made high-resolution text-to-image synthesis practical on hardware that could not comfortably run pixel-space diffusion at the same scale.
Why it mattered then
When latent diffusion models appeared, diffusion had already shown strong sample quality, but pixel-space training and sampling at high resolution demanded substantial compute. Moving the generative loop into a learned latent space cut the spatial footprint of every denoising step without abandoning the diffusion training objective that had proven reliable. Cross-attention conditioning further meant text prompts—and other signals—could steer the same UNet backbone rather than requiring a bespoke architecture per modality. The result was a practical path to high-resolution conditional image synthesis that laboratories and later open releases could actually run.
Why it matters now
The same split—compress, diffuse in latent space, decode—still underlies widely used open image generators descended from this line. Understanding where the UNet operates, how text enters, and what the autoencoder is responsible for clarifies trade-offs people still meet daily: resolution versus speed, prompt fidelity versus latent capacity, and why changing the VAE or the text encoder is a different kind of intervention from changing sampler steps. The architecture is no longer only a research result; it is the mental model behind a large share of local and hosted image tooling.
The surprising detail
The generative model never paints pixels during its iterative loop. All the careful denoising happens on a compressed tensor; the familiar RGB image is produced only once, by a decoder that was trained separately and then frozen for the diffusion stage. High-resolution output is therefore less a triumph of a bigger denoiser than of refusing to denoise at full spatial size until the very last moment.
What is disputed
Public discussion often treats “Stable Diffusion” as a single fixed network. The latent-diffusion paper describes a family of models and conditioning mechanisms; production checkpoints differ in autoencoder, text encoder, and training data. Claims about exact latent downsampling factors or parameter counts should be tied to a specific release, not assumed from the paper title alone.
Remember this
Stable Diffusion-style models denoise in an autoencoder’s latent space and inject text through cross-attention in the UNet; pixels appear only at decode time.
Test yourself
A team wants sharper fine texture in outputs and proposes training the diffusion UNet at full pixel resolution while keeping the same text cross-attention design. What cost does latent diffusion specifically avoid, and what would they give up by leaving latent space?
They would give up the main efficiency gain of the architecture: every denoising step would run on the full pixel grid instead of a spatially smaller latent, so compute and memory per step rise sharply with resolution. Latent diffusion avoids that by confining the iterative UNet work to the compressed representation and using the decoder only once at the end. Keeping cross-attention does not restore that saving; the saving comes from where the UNet operates, not from how text is injected.
Go deeper
- [2112.10752] High-Resolution Image Synthesis with Latent Diffusion Models · arxiv.org
- [2208.04201] Content-Based Landmark Retrieval Combining Global and Local Features using Siamese Neural Networks · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.