II · THE IDEA · ARTIFICIAL INTELLIGENCE
Encoder, Decoder, or Both
▶ Listen · narrated
The transformer arrived as a single idea in 2017, but within two years split into three distinct shapes. Each was built for a different task, and only one scaled into chatbots.
At a glance
- Encoder-only
- BERT. Sees entire input at once. Built for classification and understanding tasks.
- Encoder-decoder
- T5. Input processed bidirectionally, output generated left-to-right. Built for translation and summarisation.
- Decoder-only
- GPT. Processes input and generates output in one left-to-right pass. Built for open-ended generation.
Think of three ways to read a sentence. The first way: you see the whole sentence at once, then decide what it means. That is BERT, an encoder-only model. It is excellent at understanding and categorising, but it cannot write a continuation because it was never designed to produce text one word at a time. The second way: you read the sentence carefully, then write a response, keeping the original in mind as you go. That is T5, an encoder-decoder model. It separates understanding from generation, which works well for tasks like translation where input and output are distinct. The third way: you read from left to right, and as you read, you start writing, treating the whole thing as one continuous flow. That is GPT, a decoder-only model. It does not separate reading from writing—it just continues the sequence. This last approach turned out to scale best for conversations, because a conversation is already a sequence, and the model simply keeps it going.
The three architectures differ in their attention masks and in whether they separate encoding from decoding. BERT uses bidirectional self-attention across the entire input: each token attends to every other token, which allows rich contextual representations but provides no mechanism for autoregressive generation. BERT is trained with masked language modelling—random tokens are hidden and the model learns to predict them from context—and it produces a contextualised embedding for each input token, suitable for classification or span selection.
T5 uses the full encoder-decoder structure from the original transformer. The encoder applies bidirectional attention to the input, and the decoder applies causal attention to its own growing output while also cross-attending to the encoder's final states. This separation allows the model to fully process the input before beginning generation, which is efficient for tasks where the input is fixed and the output is a bounded transformation of it. Training uses a span-corruption objective, a generalisation of masked language modelling, and every task is framed as text-to-text.
Decoder-only models like GPT use only causal self-attention: each token attends to itself and all previous tokens, but never to future tokens. The prompt and the generated response are part of the same sequence, and the model is trained to predict the next token at every position. This means the same mechanism handles both understanding the prompt and generating the response. Inference is a single forward pass per generated token, with no separate encoding step. The architecture scales well because there is only one attention pattern, only one set of layers, and the training objective—next-token prediction—is simple and parallelisable. The trade-off is that the model cannot look ahead, so reasoning that requires considering multiple future possibilities must be learned implicitly or handled through techniques like sampling multiple continuations.
Look closer
BERT cannot generate text fluently
BERT's encoder layers use bidirectional attention—each token can attend to every other token in the input, including those to its right. This makes it excellent at understanding context for classification or filling in a single masked word, but it has no mechanism for generating a sequence one token at a time. You cannot ask BERT to write the next sentence, because its architecture was not designed to produce open-ended continuations. It reads; it does not write.
T5 treats everything as text-to-text
T5 uses the full encoder-decoder structure: the encoder processes the input bidirectionally, then the decoder generates the output autoregressively, attending both to its own growing output and to the encoder's representation of the input. This separation makes T5 well-suited to tasks with a clear input and output, like translation or summarisation. The input is fully understood before generation begins, which is elegant for bounded tasks but adds architectural complexity.
Decoder-only models do both jobs with one stack
GPT and its successors use only decoder layers, processing everything—the prompt and the response—as a single left-to-right sequence. The model attends to all previous tokens, so it understands context, but it never looks ahead, so it can generate fluently. This unified structure turned out to scale more predictably than encoder-decoder models, and it maps naturally onto the conversational format: the entire dialogue history is simply more input tokens, and the model continues the sequence. Simplicity won.
The story
When the transformer architecture arrived in 2017, it had two halves: an encoder that processed the input and a decoder that generated the output. Both used self-attention, but the encoder could attend to the entire input at once, while the decoder attended only to tokens it had already generated, plus the encoder's output. This asymmetry made sense for translation, the task the architecture was introduced to solve.
Within two years, researchers had pulled the transformer apart and reassembled it in three distinct configurations. BERT, published in 2018, kept only the encoder. It was trained to predict masked words in a sentence, a task that required understanding context from both directions. BERT could not generate text in the way we now expect from a language model, but it excelled at classification, question answering, and any task where the goal was to understand a fixed input rather than produce an open-ended output.
T5, published in 2019, kept the full encoder-decoder structure but reframed every task as text-to-text: translation, summarisation, question answering, even classification, all treated as problems where you read an input and write an output. This unified framing was intellectually satisfying and produced strong results, but it required maintaining two separate stacks of layers and managing the handoff between them.
GPT kept only the decoder. It processed everything as a single left-to-right sequence, attending to all previous tokens but never looking ahead. This meant it could both understand context and generate text, using the same mechanism for both. The prompt was simply the beginning of the sequence, and the model's job was to continue it. There was no separate encoding step, no handoff between components, just one stack of layers doing one thing repeatedly.
The architectural difference had practical consequences. Decoder-only models scaled more predictably as they grew larger. They mapped naturally onto open-ended generation tasks, including conversation, where there is no clear boundary between input and output—just an ongoing exchange that the model continues. Training was simpler because there was only one attention pattern to optimise, and inference was faster because there was no encoder to run separately.
By the time large language models became capable enough to converse fluently, the decoder-only architecture had won. BERT remains important for classification and embedding tasks, and encoder-decoder models still appear in specialist applications, but when you chat with a model today, you are almost certainly talking to a decoder-only system. The shape that seemed least general in 2019 turned out to be the most versatile.
Why it mattered then
Each architecture was a bet on what language models would be asked to do. BERT's designers believed the most important tasks were understanding and classification—deciding whether a sentence was positive or negative, whether two sentences were related, which span of text answered a question. These were the benchmarks that mattered in 2018, and BERT's bidirectional attention was well-suited to them. T5's designers wanted a single model that could handle every task, and they achieved that by treating everything as a text transformation problem. The encoder-decoder structure gave them clean separation between understanding and generation. GPT's designers were focused on generation from the start, and the decoder-only architecture was the simplest way to predict the next token. In 2019, it was not obvious that open-ended generation would become the dominant use case, or that a single architecture optimised for continuation would turn out to be flexible enough for everything else.
Why it matters now
The decoder-only architecture now underpins nearly every conversational AI system in wide use. ChatGPT, Claude, Llama, and their successors are all decoder-only models, processing prompts and responses as a single unbroken sequence. The architectural choice matters because it determines what is easy and what is hard: decoder-only models generate fluently and scale predictably, but they cannot look ahead, which makes certain reasoning patterns more difficult to learn. BERT-style encoders still appear in retrieval systems and embedding models, where understanding a fixed input matters more than generating a response. T5-style encoder-decoder models persist in translation and summarisation, where the input and output are genuinely separate. But the general-purpose language model, the one expected to do a bit of everything, is now almost always decoder-only. The architecture that won was not the most expressive or the most theoretically elegant—it was the one that scaled.
The surprising detail
The decoder-only architecture was not initially seen as the most promising. BERT had achieved state-of-the-art results on nearly every benchmark that mattered in 2018, and its bidirectional attention seemed like a clear advantage for understanding language. Early GPT models were impressive at generation but weaker at the classification and question-answering tasks that dominated academic evaluation. The shift happened not because decoder-only models became better at those tasks, but because the tasks people cared about changed. Once generating coherent multi-turn conversations became the goal, the architecture that had seemed narrowly specialised turned out to be the one that generalised. The lesson, in retrospect, is that architectural flexibility matters less than alignment with the task you will actually scale toward—and in 2018, almost no one predicted that task would be open-ended dialogue.
Remember this
The architecture determines what the model finds easy. Decoder-only won because continuation turned out to be the task that mattered most.
Test yourself
You have a fixed compute budget and need to build a model for two tasks: classifying whether a support ticket is urgent, and drafting a reply to it. You can train one large decoder-only model, or one smaller BERT-style encoder and one smaller decoder-only generator. What is the architectural trade-off, and when might each approach be better?
The single large decoder-only model can do both tasks, but it will be slower and more expensive at inference time because it generates tokens autoregressively even for the classification task, where you only need a single label. The split approach lets you use the fast, efficient encoder for classification and invoke the generator only when you need to draft a reply. This is cheaper at inference if most tickets do not need a drafted response. However, the split approach requires maintaining two models, and the smaller size of each may hurt quality on both tasks compared to one large model. The decoder-only approach is simpler and more flexible if the tasks blur together—if, for instance, you want the classification to be justified in generated text, or if the boundary between understanding and responding is not sharp. The choice depends on whether your tasks are genuinely separate and whether inference cost or model simplicity matters more.
Go deeper
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · arXiv · Jacob Devlin et al. · 2018-10-11
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer · arXiv · Colin Raffel et al. · 2019-10-23
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.