II · THE IDEA · ARTIFICIAL INTELLIGENCE
Video Diffusion with Temporal Consistency
▶ Listen · narrated
A plausible still is easy; a second later that still must still look like the same world. Video diffusion is largely the craft of enforcing that constraint.
At a glance
- Core problem
- Frames must stay coherent in motion, identity and lighting
- Architecture
- Space–time U-Nets with temporal attention or convolutions
- Imagen Video
- Cascaded base, spatial and temporal super-resolution models
- Text pathway
- Frozen T5 encoder conditioning the diffusion cascade
- Output target
- High-definition video at 24 frames per second
Think of asking several painters to illustrate a flipbook one page at a time, each without seeing the others’ pages. The drawings may be lovely, yet the character’s coat changes colour and the lamp jumps around the room. Video diffusion is closer to one workshop working on the whole flipbook at once: while cleaning noise off page five, the painters can look at pages four and six.
Two further tricks help. First, the workshop sketches a small, slow version of the flipbook before anyone adds fine detail—so the big motions are agreed early. Second, separate passes thicken the pages (more detail in each drawing) and add in-between pages (smoother motion). Words from a text encoder are pinned to the wall for every pass, so the story stays aligned from the rough sketch to the finished book.
That is the plain idea behind Video Diffusion Models and Imagen Video: denoise frames together in time, and build the clip from coarse to fine rather than hoping separately painted stills will agree.
Video Diffusion Models define a diffusion process on a video tensor and train a 3D U-Net denoiser whose residual blocks include temporal convolution or attention alongside spatial operators. The reverse process therefore predicts noise (or velocity) with cross-frame receptive fields, enabling joint conditional and unconditional video generation. Reconstruction guidance can further bias samples toward given reference frames by combining the denoiser score with a reconstruction gradient.
Imagen Video factors generation into a cascade. A base video diffusion model samples a low-resolution, low-frame-rate clip conditioned on frozen T5 text embeddings. Spatial super-resolution diffusion models upsample frames, conditioned on text and on the upsampled conditioning video. Temporal super-resolution diffusion models increase frame rate by synthesising intermediate frames, again with text and coarse-video conditioning. The published system targets high-definition output at 24 fps. Consistency emerges from (1) temporal mixing inside each denoiser and (2) hard conditioning of each stage on the coarser clip, which prevents later stages from freely re-sampling global identity and layout.
Limitations follow from the same design: compute and memory scale with the joint space–time volume; cascade errors propagate upward; and temporal modules only enforce coherence within the clip length and inductive bias they were trained on. Parallel independent image samples plus post-hoc warping lack both the joint score and the inherited coarse commitment, which is why they typically lose on identity and lighting stability.
Look closer
Joint modelling over the whole clip
Video Diffusion Models treat a clip as one tensor and learn a diffusion process over all of its pixels together. A 3D U-Net mixes information across height, width and time in the same network, so denoising one frame is never done in isolation from its neighbours. Temporal layers sit beside the familiar spatial ones; the reverse process therefore has a path through which motion and appearance can stay aligned as noise is removed.
Cascades instead of one giant leap
Imagen Video does not ask a single network to invent high-resolution, long-horizon video at once. A base video model produces a low-resolution, low-frame-rate clip; spatial super-resolution models add detail within each frame; temporal super-resolution models insert frames between existing ones. Each stage is itself a diffusion model, specialised to a narrower job, and each inherits conditioning from the text and from the coarser video already generated.
Conditioning that reaches every stage
Text is encoded once by a frozen T5 model and then fed throughout the cascade. The base model and the super-resolution models all see that signal, so wording about motion, subject and style can influence both the coarse trajectory and the fine spatial and temporal detail. Reconstruction-style guidance, explored in the earlier video diffusion work, further ties generated content to provided reference frames when those are available.
The story
Diffusion models learned to turn noise into images by reversing a gradual noising process. Extending that idea to video is not only a matter of stacking more pixels. A clip is a sequence: the same object should keep its shape, a shadow should move with the light, and a walking figure should not drift into a different person halfway through. Independent image samples, however good each frame looks alone, fail those tests almost immediately.
Video Diffusion Models approach the problem by defining the generative process over the full video tensor. The denoiser is a U-Net whose blocks operate in space and time together. Convolutions and attention that look along the temporal axis let each frame’s estimate depend on the others, so the reverse process is jointly constrained. The same framework supports unconditional generation and conditioned settings, including text and, where useful, guidance that pulls samples toward known frames through a reconstruction gradient.
That joint formulation already improves coherence relative to frame-wise generation, but high definition and longer clips strain a single network. Imagen Video answers with a cascade. A base video diffusion model first samples a low-resolution, low-frame-rate sequence conditioned on text embeddings from a frozen T5 encoder. Spatial super-resolution diffusion models then increase per-frame resolution, still conditioned on text and on the lower-resolution video. Temporal super-resolution models raise the frame rate by synthesising intermediate frames. The final system is aimed at high-definition video at twenty-four frames per second, built as a stack of specialised denoisers rather than one monolithic leap from noise to finished film.
Temporal consistency is therefore not a post-process. It is baked into how information moves inside the network and how responsibility is divided across the cascade. Coarse motion and layout are settled early, when the clip is cheap to represent; later stages refine texture and insert frames while remaining anchored to what the earlier stages already committed to. Failures of identity, lighting drift and stuttering motion are reduced not by a separate tracker, but by giving the denoiser explicit pathways across time and by never asking one model to solve every scale at once.
The design choice is conservative in one sense and ambitious in another. Conservative, because it reuses the diffusion machinery and the U-Net backbone that already worked for images. Ambitious, because the sample is now a four-dimensional object and every conditioning signal—text, coarser video, optional reference frames—must be threaded through multiple trained stages without breaking the alignment those stages are meant to protect.
Why it mattered then
By 2022, image diffusion had become a practical route to high-quality stills, and the obvious next pressure was motion. Naïve extensions—generating each frame alone, or animating with weak coupling—produced flicker, identity drift and physically implausible jumps. Video Diffusion Models gave a clean probabilistic framing: treat the clip as the object being denoised, and put temporal computation inside the U-Net so coherence is part of the likelihood the model learns. Imagen Video then showed that the cascaded recipe already used for large image systems could be carried over to video, with separate spatial and temporal super-resolution stages and a strong frozen language encoder. Together they marked a shift from “make frames and hope” to architectures whose inductive bias is explicitly temporal.
Why it matters now
Modern video generators still rest on the same two ideas: couple frames inside the denoiser, and factor hard generation into coarser and finer stages. When a system keeps a face consistent across a pan, or inserts frames without introducing rubbery motion, it is usually exercising some descendant of space–time blocks and cascaded refinement. Understanding those choices helps when reading new architectures, judging failure modes, or deciding where compute should go—more temporal capacity in the base model, or more careful super-resolution later. The 2022 papers remain a clear map of the problem rather than a closed historical chapter.
The surprising detail
Part of the coherence gain does not come from a clever motion module at all, but from refusing to generate the finished clip in one shot. By locking large-scale structure into a cheap, low-rate base sample and only then asking other models to add pixels and frames, Imagen Video turns temporal consistency into a constraint inheritance problem: later stages are not free to reinvent the world, only to elaborate a world already sketched. The cascade is as much a consistency device as a resolution device.
What is disputed
These papers establish architectural patterns and empirical gains for temporal coherence; they do not claim a single best coupling mechanism. How much consistency should come from joint 3D denoising versus cascaded conditioning remains a design choice that later systems continue to revisit, and reported quality depends on data, compute and evaluation protocol as much as on the block diagram.
Remember this
Video diffusion keeps frames coherent by denoising them jointly in time and by refining a coarse clip through cascaded spatial and temporal stages, not by painting stills and stitching them later.
Test yourself
A team proposes generating each video frame with a strong image diffusion model, then smoothing the sequence with an optical-flow warping pass. Using only the architectural ideas in Video Diffusion Models and Imagen Video, name two distinct reasons this is likely to underperform a space–time cascade on identity and lighting consistency.
First, the image model never sees neighbouring frames while denoising, so nothing in its reverse process couples identity, colour or illumination across time; flow warping can only rearrange what was already sampled independently, and cannot recover agreement that was never generated. Second, a cascade settles coarse layout and motion in a base video model and then conditions super-resolution on that video, so later detail is anchored to an existing temporal skeleton—whereas post-hoc warping has no equivalent inherited commitment and must impose consistency after the hard generative choices are already fixed.
Go deeper
- [2204.03458] Video Diffusion Models · arxiv.org
- [2210.02303] Imagen Video: High Definition Video Generation with Diffusion Models · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.