Vector-Quantized Variational Autoencoders (VQ-VAE)

A standard variational autoencoder compresses an input into a continuous latent distribution and reconstructs from a sample of that distribution. The continuous latent is convenient for gradient descent, yet it is an awkward partner for the kinds of prior that generate sharp, structured samples. PixelCNN-style models, for example, are natural over discrete symbols; over continuous vectors they are less so. Generative adversarial networks, by contrast, can produce crisp images, but they do not give a straightforward likelihood or a stable latent code one can edit and reuse.

Vector-quantised variational autoencoders attack that gap directly. The architecture keeps the encoder–decoder shell of a VAE, but inserts a learned codebook between the two. The encoder maps the input to a grid of continuous vectors. Each vector is replaced by the nearest prototype in a shared table of embeddings. The decoder sees only those chosen embeddings and must reconstruct the original signal — image, audio waveform, or other modality considered in the work. The latent that matters is therefore an array of discrete indices into the codebook, not a cloud of continuous coordinates.

Because nearest-neighbour selection has no useful gradient, training uses a straight-through estimator: decoder gradients are copied back to the encoder as if the quantisation step were the identity. The codebook itself is updated by a codebook loss that moves the selected embedding toward the encoder output. A commitment loss pulls the encoder the other way, discouraging it from wandering far from the table. Together with ordinary reconstruction error, these three terms keep encoder, codebook and decoder aligned without requiring an adversarial discriminator.

Once the autoencoder is trained, generation is a second stage. A prior is fitted to the discrete codes observed on the training set. In the original paper that prior is a PixelCNN-family model over the map of code indices. Sampling from the prior yields a field of codes; the frozen decoder turns those codes into an image or a waveform. The discrete bottleneck is what makes this second stage practical: the prior operates over a finite vocabulary rather than over continuous vectors.

VQ-VAE-2 pushes the same idea further by stacking codebooks at more than one resolution. A top-level codebook captures global structure; a lower-level codebook, conditioned on the top codes, fills in local detail. Priors are trained in stages to match that hierarchy. The result is a likelihood-oriented path to diverse, high-fidelity images that still rests on a learned discrete representation rather than on adversarial training alone.

The bridge the editorial angle names is therefore architectural, not rhetorical. Likelihood-based models gain a latent that is easy to model with powerful discrete priors; the reconstructions need not pay the usual continuous-VAE blur tax, because the decoder is fed sharp codebook vectors rather than noisy samples from a diagonal Gaussian. Whether that fully closes the gap with GANs depends on the prior and the codebook size — the papers are careful not to claim a universal win — but the discrete codebook is the concrete mechanism that makes the comparison possible.