II · THE IDEA · ARTIFICIAL INTELLIGENCE
Diffusion and Image Generation
▶ Listen · narrated
A photograph emerges from television snow. Not by guessing pixels one by one, but by learning what a small step toward clarity looks like, then taking that step a thousand times.
At a glance
- Training task
- Given a clean image with added noise, predict the noise
- Generation method
- Start from random noise, denoise iteratively
- Typical steps
- 50 to 1000 denoising iterations per image
- Latent diffusion
- Operates in compressed representation space, not pixel space
Imagine you have a photograph, and you slowly add television static to it until nothing recognisable remains — just random noise. A diffusion model learns to reverse that process. During training, you show it thousands of photographs at various stages of corruption and teach it to recognise what the noise looks like at each stage. It never learns to draw. It only learns to spot and subtract noise.
Generation works by running that skill backward. Start with pure static. Ask the model: if this were a noisy photograph, what would the noise component be? Subtract a little of that prediction. Now the static is slightly less random. Repeat hundreds of times, and structure emerges. First vague shapes, then recognisable objects, then fine details. The model is taking small, reliable steps toward clarity, using only its learned ability to distinguish signal from noise. By the final step, the noise has been refined into a coherent image. This is why generation is slow — you cannot skip steps. The model never learned to make large leaps, only to clean up what is already almost there.
A diffusion model is trained to predict the noise component of a corrupted image, conditioned on a timestep that indicates the noise level. The forward process is a fixed Markov chain: at each step t, Gaussian noise is added according to a variance schedule, until the image becomes indistinguishable from an isotropic Gaussian. The model learns the reverse process — a parameterised denoising function that estimates the noise added at timestep t, given the noisy image and t itself.
Training uses a simplified objective derived from the variational lower bound. For each training sample, you sample a random timestep, add the corresponding amount of noise to the clean image, and train the model to predict that noise. The loss is typically mean squared error between predicted and actual noise. The model is usually a U-Net with attention layers, conditioned on the timestep via sinusoidal embeddings and on text prompts via cross-attention.
Generation starts from a sample of pure Gaussian noise. At each step, the model predicts the noise component, and you update the sample by subtracting a scaled version of that prediction and adding a smaller amount of new noise (except at the final step). The update rule comes from the learned reverse transition kernel. Sampling requires evaluating the model once per timestep — typically 50 to 1000 times. Various samplers (DDIM, DPM-Solver, others) reparameterise the updates to allow fewer steps with acceptable quality loss, but all depend on the same learned denoising function.
Latent diffusion applies this process in the latent space of a pretrained autoencoder, reducing the dimensionality by a factor of 16 to 64. The autoencoder is frozen during diffusion training. This compression makes high-resolution generation tractable but introduces a quality ceiling determined by the autoencoder's reconstruction fidelity. Text conditioning is handled by encoding prompts with a separate language model (often CLIP or a T5 variant) and injecting the embeddings via cross-attention at each U-Net layer.
Look closer
The training data is pairs of images: clean and deliberately corrupted
Take a photograph. Add Gaussian noise — the kind that looks like static. The model's job is to predict what noise was added, given the noisy result and a number indicating how much corruption was applied. You do this with many noise levels, from barely perceptible to complete obliteration. The model never tries to generate during training. It only learns to recognise and subtract noise. Generation is what happens when you run that learned skill backward, starting from pure randomness.
Each denoising step is small, and the model knows how noisy the input still is
The model receives two things: the current noisy image and a timestep value that encodes how far through the denoising schedule it has progressed. Early steps work on almost pure noise, where only the coarsest structure can be inferred. Later steps refine details in a nearly clean image. The same network handles all timesteps, conditioned on that scalar input. This is why generation is slow — you cannot skip from noise to photograph in one pass. The model learned to take small, reliable steps, not large guesses.
Latent diffusion moves the entire process into a compressed space
Running diffusion directly on megapixel images is computationally prohibitive. Rombach and colleagues trained a separate autoencoder to compress images into a much smaller latent representation — typically sixty-four times fewer values — then ran the diffusion process there. The final latent is decoded back into pixels only once, at the end. This is the architecture behind Stable Diffusion. The trade-off is that fine detail must survive the round trip through the autoencoder, and some high-frequency information is lost or altered in compression.
The story
Diffusion models generate images by learning to remove noise, then running that process in reverse. The principle is completely unlike next-token prediction. There is no autoregressive sweep, no vocabulary, no left-to-right order. Instead, the model is trained on a simple task: given a clean image with added noise, predict what noise was added.
During training, you take real photographs and corrupt them with Gaussian noise at varying levels of severity. A scalar timestep tells the model how noisy the input is. The model learns a function that estimates the noise component, and the training loss penalises errors in that estimate. You never ask it to generate anything. You only teach it to recognise corruption.
Generation inverts the process. Start with an array of random noise — pure static, no structure. Feed it to the model along with a timestep indicating maximum noise. The model predicts what portion of that array is noise, and you subtract a small fraction of that prediction. The result is slightly less noisy. Repeat, decrementing the timestep each time, and structure begins to emerge. After hundreds of steps, the noise has been refined into a coherent image.
The mathematics come from thermodynamics. Ho and colleagues formalised the process as a Markov chain with learned reverse transitions, where the forward process gradually destroys information and the reverse process reconstructs it. The training objective derives from variational inference, though in practice the model simply learns to predict noise at each corruption level.
Text conditioning is added by encoding a prompt into a vector and injecting it into the denoising network, typically through cross-attention layers. The model learns associations between language and visual structure during training on captioned images. At generation time, the text steers which image emerges from the noise, though the iterative denoising process remains the same.
Latent diffusion, introduced by Rombach and colleagues, moves the entire process into a compressed representation. A separate autoencoder is trained first, learning to map images into a lower-dimensional latent space and back. The diffusion model then operates in that latent space, where arrays are much smaller and computation is faster. The final latent is decoded into pixels only once. This architecture made high-resolution generation practical on consumer hardware, though it introduced a dependency on the autoencoder's fidelity. Fine textures and small text often degrade in the compression round trip.
Why it mattered then
Ho and colleagues published their denoising diffusion probabilistic models in 2020, at a moment when generative adversarial networks still dominated image synthesis despite well-known training instabilities. Diffusion offered a different trade: stable training and high sample quality in exchange for much slower generation. The iterative denoising process was computationally expensive, but it avoided the mode collapse and discriminator pathologies that plagued GANs. Early diffusion models were slow enough that practical deployment seemed distant, but the training procedure was robust and the results were often sharper than GAN outputs at comparable resolutions. Rombach's latent diffusion work in 2022 moved the bottleneck. By compressing images before applying diffusion, generation time dropped by an order of magnitude, and memory requirements fell enough that a capable model could run on a single consumer graphics card. That shift made diffusion models deployable outside research labs, and within months they were being incorporated into commercial tools and open-weight releases.
Why it matters now
Diffusion models are now the dominant architecture for image generation, displacing GANs almost entirely in production systems. The iterative denoising process allows fine-grained control: you can stop partway, edit the latent, and continue, or you can inject structure at specific timesteps. Text-to-image models built on latent diffusion underpin widely deployed tools, and the same principles have been extended to video, audio, and three-dimensional shape generation. The architecture's stability during training means that scaling up — more parameters, more data, longer training — reliably improves output quality, which was not true for GANs. The cost is generation speed. Even with latent compression and optimised samplers, producing a single image still requires dozens of network evaluations, and real-time generation remains difficult. Research continues on faster sampling schedules and distilled models that approximate the full denoising trajectory in fewer steps, but the fundamental trade-off between quality and iteration count has not been eliminated.
The surprising detail
The forward noising process is never learned. It is a fixed schedule, chosen in advance, that gradually adds Gaussian noise until the image is indistinguishable from random static. The model learns only the reverse. This asymmetry means that generation depends on a hand-designed noise schedule, and different schedules can produce noticeably different results even with the same trained model. Researchers have explored learned schedules, but most deployed systems still use manually tuned curves, and there is no consensus on the optimal shape. The choice of schedule interacts with the number of sampling steps in ways that are not fully understood, and practitioners often adjust both together when balancing quality against speed.
Remember this
Diffusion models generate by learning to denoise, then starting from noise. The principle is iterative refinement, not sequential prediction.
Test yourself
You have a diffusion model trained on images at 512×512 resolution. You want to generate a 1024×1024 image. Explain why you cannot simply run the model on a larger noise array, and describe what actually happens if you try.
The model learned spatial relationships and structure at a specific scale. Its convolutional layers and attention patterns expect features to appear at certain positions and frequencies corresponding to 512-pixel images. If you feed it a 1024×1024 noise array, those learned priors misalign with the actual content. Fine details may be generated at the wrong scale, repeated patterns can emerge, and the overall composition often looks like four 512×512 tiles stitched together rather than a coherent large image. Some models can extrapolate slightly beyond their training resolution, but the quality degrades quickly. Proper high-resolution generation requires either training at that resolution from the start, or using techniques like cascaded models that upsample in stages, or methods that tile and blend overlapping regions.
Go deeper
- Denoising Diffusion Probabilistic Models · arXiv · Jonathan Ho et al. · 2020-06-19
- High-Resolution Image Synthesis with Latent Diffusion Models · arXiv · Robin Rombach et al. · 2021-12-20
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.