Contrastive Learning Across Vision, Audio and Text

Start with the problem. You want a model that connects what a thing looks like, what it sounds like and what it is called. The obvious route is to pay people to write labels: this clip contains a violin, this one contains applause. Labelling by hand is slow and expensive, and it caps what the model can ever say, because it can only learn the words that appear on the label list. Contrastive learning takes a cheaper route. Video from the web already arrives as a bundle: the frames, the soundtrack, and often a transcript of whatever was said. All 3 were recorded at the same moment. Nobody had to assert that they belong together; the recording itself did. The method turns that accidental togetherness into the training signal.

The machinery begins with encoders. An encoder is a network that turns one kind of input into an embedding — a fixed-length list of numbers that works like an address in an abstract space. There are 3 encoders here, one per kind of input, and internally they share almost nothing. Frames are a grid of pixels changing over time; sound is a waveform; a caption is a sequence of word pieces. Each encoder can be built however suits its material. The only firm requirement is that all 3 produce lists of the same length, because 2 lists of the same length can be compared: the closer they are, the more the model treats their inputs as belonging together. That shared output size is the joint space. Everything the model comes to know about the link between a hiss and the word simmer lives in the geometry of that space — in what sits near what — not inside any single encoder.

Training is a matching game played on batches. Take a few hundred clips at once. Compute every clip's video, audio and text embeddings. Now lay the video embeddings down the side of a grid and the caption embeddings across the top, and fill every cell with a similarity score. The cells on the diagonal — clip one's video against clip one's caption, clip two's against clip two's — are the true pairs. Every other cell is a mismatch, and here is the economy of the whole method: nobody had to label those mismatches. Clip three's caption is automatically a wrong answer for clip one's video, simply because they came from different clips. Training means nudging the encoders so the diagonal scores rise and all the rest fall. A batch of a few hundred clips hands the model a few hundred wrong answers for every right one, free of charge.

The word contrastive names the pushing as well as the pulling, and the pushing is not optional. Pulling alone has a perfect, useless solution: send every input — every frame, every sound, every sentence — to the identical list of numbers. Then every true pair matches exactly, the training score is perfect, and the model has learned nothing, because it can no longer tell anything apart. This failure has a name, representation collapse. The mismatched pairs are what forbid it. Because the model is also punished whenever an impostor scores high, it cannot flatten everything into one point. It must hold different things apart while holding matched things together, and a space that does both is a space where distance carries meaning: near means related, far means unrelated.

One dial governs how harsh the game is. The loss compares each true pair's score against the impostors' scores, and a scale factor — usually called the temperature — decides how sharply. A low temperature punishes a near-miss almost as hard as an absurd guess, which carves tightly separated clusters but can make training unstable. A high temperature is forgiving and yields a blurrier space. There is no known correct value; published choices vary, and the setting is a genuine trade-off rather than a solved detail.

Going from 2 kinds of input to 3 is less tidy than it sounds. Video, audio and text give you 3 pairings to satisfy at once, and they are not equally trustworthy. A soundtrack really is simultaneous with its frames; the microphone and the camera ran together. Speech often is not. A narrator says now let it rest for 10 minutes and then, seconds later, does something else on screen. The work on uncurated instructional video [1912.06430] uses exactly this material — how-to video taken from the web as found, not assembled in a laboratory — and treats the loose fit between speech and action as part of the problem to solve, not a detail to ignore. That admission is worth holding on to: any tidy summary of these systems that assumes the transcript describes the picture is describing cleaner data than the models actually saw.

There is a second cost, quieter but real. Contrastive training is hungry for clips and for computing power, and part of the hunger comes directly from the mechanism. The training signal is stronger when there are many impostors to rule out at once, which pushes towards large batches, which demand large machines and large collections of video. The compact transformer paper [2104.05704] sits at the other end of that tension. It is explicitly about escaping what it calls the big data paradigm — getting transformer architectures to learn usefully from modest datasets rather than assuming enormous ones. The 2 papers are not about the same task, but together they mark the ends of a real trade-off. One shows how much can be learned when supervision is free and data is nearly unlimited. The other asks what to do when it is not.

If the training works, what you end up with is not a classifier — not a machine that sorts inputs into a fixed list of categories. It is a measuring device. Give it any image, any sound and any sentence, and it reports how close they sit in the shared space. That single ability supports everything else. Finding a photo by typing a description is a nearness query: your words become an embedding, and the nearest photos win. So is zero-shot classification, where the model labels an input it was never trained to categorise by embedding candidate sentences — a photo of a dog, a photo of a cat — and picking the nearest. None of these abilities was programmed in. Each falls out of the geometry the matching game produced.