II · THE IDEA · ARTIFICIAL INTELLIGENCE
Speech Recognition and Synthesis
▶ Listen · narrated
The same architecture that translates between languages can turn sound into text, but only after learning which fragments of a waveform correspond to which fragments of meaning, with no explicit labels to guide it.
At a glance
- Recognition task
- Map acoustic signal to text sequence
- Synthesis task
- Map text sequence to acoustic signal
- Training regime
- Weakly supervised on internet audio, or self-supervised masking of unlabelled speech
- Known failure mode
- Hallucinating plausible text during silence or background noise
Imagine you are trying to learn a language by watching videos with subtitles, but the subtitles only tell you what was said in each scene, not exactly when each word was spoken. You would learn to match speech sounds with words, but you would have no reliable way to know when someone has stopped talking. Speech recognition models face the same problem. They are trained on huge amounts of audio with accompanying text — YouTube videos with captions, podcasts with transcripts — but those transcripts rarely mark the exact moment each word begins and ends. The model learns the mapping from sound to text by inferring the timing itself. This works well enough for speech, but it means the model never learns a clean boundary between speech and silence. When the audio goes quiet, the model's language component takes over and generates plausible text, because that is what it learned to do when the acoustic signal is ambiguous.
A speech recognition model converts audio into a spectrogram — a time-frequency representation where each column is a vector of frequency bins — and processes it with a Transformer encoder or similar sequence model. The output is a sequence of tokens representing text. Training can use weak supervision, where internet audio is paired with transcripts that lack precise word-level timestamps, or self-supervised pre-training, where the model learns to predict masked spans of the spectrogram from context before being fine-tuned on a smaller labelled dataset. Whisper, described by Radford et al., uses weak supervision on 680,000 hours of audio. The model learns alignment implicitly through the training objective, but because the supervision signal is weak during silence, the model does not learn a reliable silence class. During inference, when the acoustic input is quiet or ambiguous, the language model prior dominates and the model generates text with high likelihood under its learned distribution, even though no speech is present. This is a direct consequence of the training regime: the model is optimising for text likelihood given audio, and silence was not represented as a distinct, labelled state in the training data.
Look closer
Audio arrives as a spectrogram, not as samples
The raw waveform — air pressure varying thousands of times per second — is converted into a time-frequency representation before the model sees it. Each vertical slice shows which frequencies are present at that moment. The model reads this grid much as it would read a sequence of embeddings: left to right, learning which patterns in the frequency domain correspond to phonemes, words, pauses. The conversion is not learned; it is a fixed signal-processing step borrowed from decades of audio engineering.
Weak supervision means the timestamps are approximate
Whisper, the system described by Radford and colleagues, was trained on 680,000 hours of audio scraped from the internet alongside whatever subtitles or transcripts accompanied the videos. Those transcripts rarely mark the exact moment each word is spoken — they give the text for a whole segment, sometimes several sentences long. The model must infer the alignment itself during training. This works at scale, but it also means the model never learns a ground-truth boundary between speech and silence, which contributes directly to the hallucination behaviour during quiet passages.
Self-supervised pre-training masks input and predicts what was hidden
Wav2vec 2.0, described by Baevski and others, takes unlabelled speech and masks spans of the spectrogram — replacing them with a learned mask token — then trains the model to predict what was hidden. Only after this pre-training phase is a small labelled dataset used to fine-tune for transcription. The result is that most of the model's knowledge about the structure of speech comes from raw audio, not from human annotations, which makes large-scale training feasible for languages where labelled data is scarce.
The story
Speech recognition has been treated as a sequence problem since the 1980s, but until recently the dominant approach was to build separate components: one model for acoustic features, another for phonemes, another for language structure, each trained on carefully labelled data. The shift to end-to-end learning — feeding a spectrogram in at one end and reading text out at the other — became practical only when training corpora grew large enough to let a single model learn the entire mapping.
A spectrogram is a grid. Time runs horizontally, frequency vertically. Each cell holds a number representing how much energy is present at that frequency at that moment. The model reads this grid as a sequence of vectors, much as a language model reads token embeddings. The architecture is often a Transformer encoder, sometimes with convolutional layers at the front to capture local patterns in the frequency domain before the attention mechanism takes over.
The training objective is straightforward: given audio, predict the text. But there are two ways to gather training data at scale, and they lead to different failure modes. Weak supervision uses internet audio with accompanying transcripts that were not created for machine learning — YouTube captions, podcast descriptions, subtitle files. The transcripts give you the words, but not the exact moments they were spoken. The model must learn the alignment itself, which it does reasonably well for speech but poorly for silence. When the audio goes quiet, the model has learned that text usually follows audio, and it has a strong prior from its language model component about what English sentences look like, so it generates plausible words that were never said. The behaviour is not random; it is the model doing exactly what its training distribution taught it to do.
Self-supervised pre-training takes a different approach. Wav2vec 2.0 starts with unlabelled audio — just raw speech, no transcripts at all. It converts the audio to a spectrogram, masks out spans of it, and trains the model to predict the masked content from context. This is the same masking strategy used in language models, applied to the acoustic domain. Only after this pre-training phase, which can use hundreds of thousands of hours of unlabelled audio, does the system see a small labelled dataset to learn the mapping from sound to text. The advantage is that most of the model's knowledge about speech structure comes from raw audio, not from expensive human transcription. The disadvantage is that the model still depends on the quality and coverage of that final labelled dataset, and for low-resource languages the labelled data may be too small to generalise well.
Synthesis — generating audio from text — uses similar architectures in reverse, though the output is more complex. Text is discrete; audio is continuous. Most synthesis systems generate a spectrogram first, then pass it through a vocoder to produce the final waveform. The spectrogram is easier for a model to learn because it is still a grid of numbers, closer to the kinds of outputs a neural network produces naturally.
Why it mattered then
Weak supervision at scale became viable in the late 2010s because the internet had accumulated enough video with captions to train on. Before that, speech recognition depended on carefully labelled datasets like LibriSpeech, which contains hundreds of hours rather than hundreds of thousands. Those smaller datasets forced researchers to use more constrained architectures and hand-engineered features. The shift to weak supervision was not a better algorithm; it was a recognition that imperfect labels from the internet, if you have enough of them, outperform perfect labels that are too expensive to gather in quantity. Self-supervised pre-training arrived slightly later, borrowing the masking idea from BERT and applying it to audio. Both approaches reflect the same insight: labels are the bottleneck, so train on something cheaper first.
Why it matters now
Speech recognition is now embedded in devices that millions of people use daily, and the models are good enough that the remaining errors are often not recognition failures but boundary failures — the model transcribing background noise, or continuing to generate text after the speaker has stopped. These are not bugs in the usual sense. They are the model behaving consistently with a training distribution that did not cleanly separate speech from non-speech. Understanding why the model hallucinates during silence matters for anyone deploying these systems in real environments, where audio does not arrive pre-segmented. It also matters for fairness: weakly supervised models perform better on languages and accents that were common in their training data, and self-supervised pre-training only partly addresses this, because the final labelled fine-tuning dataset is still small for most of the world's languages.
The surprising detail
The hallucination behaviour during silence is not a failure of the acoustic model. It is the language model component doing its job. Whisper and similar systems are trained to predict text given audio, but they are also trained to predict plausible text given ambiguous audio, because the weak supervision regime never gave them a ground-truth silence label. When the audio is quiet, the model's prior over likely English sentences takes over, and it generates text that sounds like something a person might say. Researchers have observed the model producing the same few phrases repeatedly during long silent passages — often things like "Thank you for watching" or "Please subscribe", which are common in YouTube videos and therefore common in the training data. The model is not malfunctioning; it is interpolating in exactly the way it was trained to do.
Remember this
Speech models treat audio as a sequence, learning alignment from data rather than from explicit labels — which works at scale but leaves them unable to reliably distinguish speech from silence.
Test yourself
A weakly supervised speech model transcribes a recording that contains 30 seconds of silence in the middle. Instead of outputting nothing, it generates plausible text. Explain why this happens in terms of what the model learned during training.
The model was trained on audio paired with transcripts that gave the words for entire segments, not precise timestamps for when each word was spoken. It learned to predict text given audio, but it never learned a reliable boundary between speech and silence, because the training data did not provide one. When the audio goes quiet, the model still has a strong prior from its language model component about what English sentences look like, and it has learned that text usually follows audio in its training distribution. So it generates text that is plausible given its training data — often phrases common in the internet videos it was trained on — rather than producing no output. The behaviour is not a bug; it is the model interpolating in a region where its training signal was weak.
Go deeper
- Robust Speech Recognition via Large-Scale Weak Supervision · arXiv · Alec Radford et al. · 2022-12-06
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations · arXiv · Alexei Baevski et al. · 2020-06-20
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.