II · THE IDEA · ARTIFICIAL INTELLIGENCE
Features and Representations
▶ Listen · narrated
The single largest shift in machine learning was not a new algorithm. It was stopping telling the model what to notice and letting it work that out itself.
At a glance
- Traditional approach
- Engineer designs features; model learns only the final weights
- Modern approach
- Model learns both features and weights end-to-end from data
- Typical depth
- Multiple layers of representation, each transforming the previous
- Key advantage
- Features adapt to the task and data rather than designer intuition
Imagine you are teaching someone to recognise birds. One approach: you write a careful list of what to check — beak shape, wing pattern, size, colour — and teach them to score each item, then combine the scores. Another approach: you show them hundreds of photos with labels and let them figure out what matters. The first method is transparent and works with fewer examples, but only if your list was good. The second method requires more examples and you cannot easily see what they learned to notice, but it often ends up better because they find patterns you did not think to write down. Modern machine learning has moved from the first approach to the second. Instead of engineers designing features by hand, the model learns its own internal representations of what matters, adjusting them during training to improve performance.
In traditional machine learning pipelines, feature extraction and learning were separate stages. A domain expert would design a feature extractor — edge filters, Fourier coefficients, n-grams, hand-tuned kernels — that mapped raw input to a fixed-length vector of measurements. A learning algorithm, often convex and well-understood, then learned weights for those features. Performance was limited by the quality of the feature design, and improving it required domain insight and manual iteration. In representation learning, the model learns a hierarchy of features end-to-end via backpropagation. Each layer applies a parameterised transformation to the previous layer's output, and all parameters are optimised jointly with respect to the task loss. Early layers in a vision network might learn Gabor-like filters; middle layers learn combinations that respond to textures or parts; deep layers learn task-specific combinations. The features are distributed: information is spread across many activations rather than isolated in interpretable dimensions. This makes them harder to analyse but allows them to capture complex dependencies the designer would not have encoded. The approach requires more data and compute, and it introduces new failure modes — learned features can latch onto spurious correlations in the training set. But it scales better, because the model adapts its representation to the data and task rather than relying on a human bottleneck. Architectures like convolutional networks, residual networks and transformers are all instances of this principle: stack parameterised layers, learn all parameters together, let the intermediate representations emerge from the data.
Look closer
The same task, two architectures
In handcrafted vision systems, an engineer might write code to detect edges at various angles, then corners, then combine those into shape descriptors, then feed those fixed features into a classifier. The feature extraction is separate from learning. In a convolutional network tackling the same task, the early layers learn their own edge detectors, middle layers learn their own combinations, and late layers learn their own high-level features — all adjusted together during training. The engineer specifies the architecture but not what any filter should detect.
Features become distributed
A handcrafted feature usually corresponds to one number: edge strength at 45 degrees, or the presence of a particular phoneme. A learned representation spreads information across many numbers. A single neuron's activation may not mean anything interpretable on its own, and a single concept — "this is a dog" — may be encoded in the pattern across hundreds of activations. This makes learned features harder to inspect but allows them to capture relationships the designer never anticipated.
The representation is task-specific
If you train one network to classify dog breeds and another to detect whether a photo was taken indoors, both see the same pixels but learn different internal features. The breed classifier's middle layers might become sensitive to fur texture and ear shape; the indoor detector's to lighting angles and whether there is sky. Handcrafted features, being fixed, must serve all tasks equally poorly or well. Learned features specialise, which is why transfer learning — starting from features learned on one task — works better than starting from generic hand-designed ones.
The story
For most of the history of pattern recognition, the division of labour was clear. A human expert would study the problem, decide what mattered, and write code to measure it. Edge detectors for images. Mel-frequency cepstral coefficients for speech. Carefully normalised word counts for text. The resulting numbers — the features — went into a learning algorithm, often something simple like logistic regression, which learned only how to weight them. Performance depended heavily on the quality of the feature design, and feature design depended on domain expertise, intuition and trial and error.
This arrangement had virtues. The features were interpretable: you knew exactly what the third number meant. They often generalised across tasks in the same domain, because edges matter for many vision problems and phonemes matter for many speech problems. And they let you apply learning algorithms to problems where you had relatively little labelled data, because most of the structure was baked into the features rather than learned.
But the approach had a ceiling. The features were only as good as the designer's understanding, and for complex tasks — recognising objects in cluttered scenes, transcribing conversational speech, translating between languages — no one really knew what the right features were. You could spend years engineering a representation and still miss something crucial that a different arrangement of the same raw data would have revealed.
The alternative is to learn the features. Let the model decide, through training, what aspects of the input to measure and how to combine them. This is representation learning, and it requires a different architecture. Instead of a shallow model sitting atop fixed features, you build a deep model where each layer transforms the representation from the layer below. Early layers might learn simple patterns; deeper layers learn combinations of those patterns; the deepest layers learn combinations of combinations. The model adjusts all of these transformations simultaneously, guided by how well the final output matches the training labels.
The shift happened gradually, domain by domain, as compute and data became sufficient to train deeper networks. Convolutional networks for vision, recurrent networks for sequences, eventually transformers for language. In each case, performance on difficult tasks improved sharply once the model was allowed to learn its own features rather than relying on human-designed ones. The 2012 ImageNet result is often cited as the inflection point for vision, but the principle was older; what changed was the ability to train it at scale.
Learned representations are not always better. They require more data, more compute, and more care in training. They are harder to interpret, because the intermediate features often do not correspond to anything a human would name. And they can learn to rely on spurious patterns in the training data — shortcuts that work in the dataset but not in the world. But for tasks where the right features are not obvious, or where they are too numerous and interdependent to design by hand, learned representations have proven to be the more scalable approach. The engineer's role shifts from specifying what to notice to specifying the learning process itself.
Why it mattered then
The appeal was partly empirical and partly philosophical. Empirically, learned features kept winning. By the early 2010s, deep convolutional networks trained end-to-end were outperforming systems built on decades of careful feature engineering, often by wide margins, on standard vision benchmarks. The gap was large enough that it could not be dismissed as a quirk of one dataset or task. Philosophically, the approach aligned with a broader ambition in artificial intelligence: that the system should discover structure rather than have it encoded by a human. If you believed that human perception worked by learning hierarchical representations from data — a view with support in neuroscience — then having your model do the same felt like progress toward something more general. The fact that it also reduced the amount of domain-specific engineering required made it attractive to researchers who wanted to apply similar architectures across multiple problems without redesigning the feature extractor each time.
Why it matters now
Representation learning is now the default assumption in most areas of machine learning. When you fine-tune a language model, you are adjusting learned representations. When you train a diffusion model, it learns its own features for image structure. The shift has been so thorough that newer practitioners sometimes do not realise there was an alternative. But the questions it raises have not gone away. Learned features are opaque, and opacity becomes a problem when you need to understand why a model failed or whether it is relying on the right reasons. There is active work on interpreting learned representations, on constraining them to be more human-readable, and on hybrid approaches that combine learned and hand-designed components. The interpretability gap also matters for safety: if you cannot inspect what features a model has learned, you cannot easily check whether it has learned something dangerous or biased. Representation learning solved one problem and created another, and the second problem is still open.
The surprising detail
One unexpected consequence of learned representations is that they can be transferred. If you train a deep network on one task — say, classifying a million images into a thousand categories — the features it learns in its early and middle layers often turn out to be useful for completely different tasks, even with very little additional training. This was not obvious in advance. Handcrafted features transferred because they were designed to be general, but learned features are optimised for a specific task and dataset, so you might expect them to be narrow and brittle. In practice, they often capture broader structure than the designer would have thought to encode, precisely because they are free to learn whatever helps. This has made transfer learning a standard technique: start with a model trained on a large dataset, then adapt its learned representations to your smaller, more specific problem.
Remember this
The shift from hand-designed to learned features is the largest architectural change in modern machine learning. The model now learns what to notice, not just how to weight it.
Test yourself
You have two models trained on the same image classification task. One uses fixed handcrafted features fed into a shallow classifier. The other learns features end-to-end in a deep network. Both achieve similar accuracy on the training set. You now apply both to a related but different task. Which is likely to transfer better, and why?
The deep network's learned features are likely to transfer better, even though they were optimised for a specific task. Learned representations tend to capture hierarchical structure — early layers learn edges and textures, middle layers learn parts and patterns, late layers learn task-specific combinations. The early and middle layers often turn out to be useful for other tasks in the same domain, because they have discovered general structure in the data rather than having it prescribed. Handcrafted features transfer only if the designer anticipated what the new task would need. Learned features transfer because they adapt to whatever structure was present in the training data, which often includes regularities the designer did not know to encode. This is why transfer learning works: the model has learned a vocabulary for describing the domain, not just a solution to one problem.
Go deeper
- Representation Learning: A Review and New Perspectives · arXiv · Yoshua Bengio et al. · 2012-06-24
- representation.html · deeplearningbook.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.