II · THE IDEA · ARTIFICIAL INTELLIGENCE
Meta-Learning with External Memory
▶ Listen · narrated
Gradient updates are slow teachers. When a model must recognise a character after seeing it once, something faster than weight change is required — a place to write, and a way to read back.
At a glance
- Core idea
- An external memory lets a network store and retrieve new examples without slow weight rewriting
- MANN setup
- Controller plus addressable memory, trained across tasks so one-shot storage becomes a learned skill
- MAML contrast
- Learns an initialisation that adapts with a few gradient steps; no external store required
- Shared goal
- Fast assimilation of new tasks from very few labelled examples
- Access style
- Soft, content-based addressing rather than hard discrete lookups
Think of a clerk who already knows the filing system and a set of empty drawers that start blank each morning. The clerk’s training is long and slow: how to label a folder, which drawer to open, how to find a folder again when a similar page arrives. That training is the network’s ordinary weights, shaped across many different days and different sets of documents. The drawers are the external memory. When a single new page appears — a character, a class, a fact never seen in this form — the clerk does not rebuild the filing system. The clerk writes one folder and puts it away. The next time a matching page turns up, the clerk looks it up.
That is the memory-augmented idea. The model is trained on a stream of small tasks so that using the drawers becomes second nature. Within a task, it often has to guess a label before being told the answer, which pushes it to file what it was told and to rely on that file later. A related approach, model-agnostic meta-learning, skips the drawers. Instead it practises starting from a particular set of weights such that a few quick practice steps on a new task are enough. Same problem — learn from almost no examples — two different places to put the quick change.
Memory-augmented meta-learning couples a controller network to an external memory matrix accessed by soft, content-based addressing. During an episode the controller emits read and write operations; writes store bindings between inputs and supervised signals, and reads supply retrieved vectors that condition the current prediction. In the one-shot classification protocol, inputs are presented sequentially and the class label is delayed until after the prediction, so that later correct classification of a repeated or related input depends on successful prior writes and on accurate retrieval rather than on simultaneous label leakage.
Meta-training samples tasks from a distribution. Slow weights — the controller parameters and the addressing machinery — are optimised across tasks for episode-level performance. Fast state lives in the memory slots and is reset or overwritten between episodes. Soft addressing yields differentiable paths for end-to-end training but also interference: capacity scales with the number of locations and with how cleanly keys separate, not with a hard discrete guarantee.
Model-agnostic meta-learning (MAML) addresses the same outer objective without an external store. It optimises an initialisation θ such that a small number of inner-loop gradient steps on a task’s support set produce parameters that perform well on that task’s query set. The meta-gradient passes through the inner update, so θ is judged by post-adaptation loss. The method is architecture-agnostic wherever gradient descent applies; its limits are those of short optimisation trajectories and of second-order (or first-order approximate) meta-gradients.
Mechanistically the contrast is sharp. MANN-style systems perform adaptation by state writes into memory; MAML performs adaptation by parameter steps from a meta-learned start. Both assume a task distribution at train time and few labelled points at test time. Neither removes the need for that distribution to resemble the eventual deployment tasks; both can fail when support sets are unrepresentative or when the fast pathway — slots or steps — is undersized for the within-task variability.
Look closer
Prediction before the label arrives
In the memory-augmented setup, samples arrive as a sequence. The model must emit a class prediction for the current input before the true label is revealed. Only then can it write the binding of input to label into memory. That delay is deliberate: if the label arrived with the input, the network could cheat by copying rather than learning to store and retrieve. The next time a similar input appears, success depends on having written something usable and on being able to find it again.
Weights hold the strategy, memory holds the episode
Slow weights are trained across many tasks and come to encode how to use the memory — when to write, how to address, how to combine retrieved contents with the current input. Fast information about the present episode, including never-before-seen classes, lives in the external slots. The split is what makes one presentation enough: the strategy is already learned; only the contents need updating.
Two routes to the same scarcity problem
Memory-augmented networks and model-agnostic meta-learning both attack few-shot recognition, but they locate the adaptation in different places. One writes into an external store and reads back. The other starts from a carefully chosen parameter initialisation and takes a small number of gradient steps on the new task. Neither invents new labelled data; both try to make the little data that exists do more work.
The story
Ordinary supervised training assumes a long stream of examples from a fixed set of classes, and it absorbs those examples by changing weights. That process is poorly matched to a familiar human demand: see a new symbol once, then recognise it again. By the time a gradient step has moved the parameters, the moment has passed, and a single example is a weak signal for a large weight vector in any case.
Memory-augmented neural networks attack the mismatch directly. Alongside the controller network sits an external memory — a set of slots that can be written to and read from during an episode. Addressing is soft and content-based: the controller produces keys and weights over locations rather than issuing hard addresses. What gets stored is under learned control, and so is what gets retrieved. The slow weights of the controller are not asked to memorise the new class. They are asked to operate the memory.
The training regime matters as much as the architecture. Tasks are drawn from a distribution; within each task the model sees a short sequence of labelled examples, often from classes it has never met in that form before. Crucially, the label for a given sample is withheld until after the model has made its prediction. The only reliable way to succeed on a later repeat of a similar input is to have written the earlier binding into memory and to find it again. Across many such tasks the controller learns a general procedure for one-shot storage and retrieval. At test time the same procedure is applied to fresh classes. The paper that introduced this framing for memory-augmented nets demonstrated the approach on problems such as character recognition from limited examples, where novel identities must be bound quickly and used at once.
A different line of work reaches the same scarcity problem without an external store. Model-agnostic meta-learning seeks an initial set of parameters such that a few gradient steps on a handful of examples from a new task produce a competent task-specific model. The meta-objective is not performance after full training, but performance after rapid adaptation. Because the update rule remains ordinary gradient descent, the method applies to a wide range of architectures; the specialisation is in the initialisation and in the bilevel training that produces it. Here the “memory” of how to adapt is folded into the starting weights rather than written into slots.
The two approaches are not rivals so much as contrasting answers to one question: where should the fast-changing information live? In an explicit memory, access is an addressing problem and capacity is bounded by the number of slots and the fidelity of soft reads and writes. In a sensitive initialisation, access is a short optimisation trajectory and capacity is bounded by how much structure that trajectory can unlock before overfitting the few shots. Both reject the idea that every new class must be absorbed by a long, from-scratch weight revision. Both treat learning-to-learn as a training distribution over tasks, not as a single fixed dataset.
What the memory-augmented view makes especially clear is the division of labour between strategy and content. Strategy — the policy for writing, addressing and reading — is expensive to acquire and is paid for across tasks. Content — the binding of this unfamiliar input to this label — is cheap to acquire if the strategy already knows where to put it and how to get it back. One-shot behaviour falls out of that division when the training tasks have forced the strategy to be good.
Why it mattered then
Deep networks of the mid-2010s were already strong when data were plentiful and classes were fixed, yet they remained awkward at the few-shot regime that matters for agents, personalisation and open-ended recognition. External memory modules, including Neural Turing Machine-style controllers, had shown that neural nets could learn to use a writable store for algorithmic tasks. Casting that store as the seat of episode-specific knowledge, and training across a task distribution with delayed labels, turned the same machinery toward one-shot classification. At the same moment, gradient-based meta-learning showed that a carefully prepared initialisation could make ordinary fine-tuning sample-efficient. Together they reframed few-shot learning as a problem of where and how adaptation is allowed to happen, rather than as a plea for more labels.
Why it matters now
Models still meet long-tailed classes, user-specific concepts and tasks that arrive with a handful of demonstrations. Fine-tuning large networks for each such case is costly and slow; pure in-context prompting shifts the burden onto context length and onto whatever structure pretraining happened to build. The memory-augmented lesson remains practical: separate stable skill from transient fact, and give the transient fact a place that can be written in one pass. The MAML lesson remains practical too: sometimes the right “memory” is a starting point in parameter space from which a few steps suffice. Modern retrieval-augmented systems, episode-level caches and rapid adapters are variations on those two placements of fast information. The scarcity problem has not disappeared; the design choice is still where the new example should live.
The surprising detail
The delayed-label curriculum is doing more work than the memory hardware alone. Without the forced gap between seeing an input and being told its class, the controller can avoid learning a general write-and-retrieve procedure. The episode protocol is not a minor implementation detail; it is the pressure that turns a writable store into a one-shot learner. Equally striking is how little the meta-learning objective cares which internal mechanism adapts: slots and soft attention on one side, a few gradient steps from a shared initialisation on the other. Same outer demand — succeed after scarce experience — two inner answers.
What is disputed
The two source papers share a few-shot goal but not a mechanism; treating memory augmentation and gradient-based initialisation as interchangeable would overstate the evidence. Reported gains are tied to particular task distributions and memory or optimisation designs, and soft addressing brings its own limits of capacity and interference that the papers characterise in their experimental settings rather than as settled universal bounds.
Remember this
Few-shot skill comes from separating strategy from content: slow weights learn how to adapt; external memory or a short gradient path holds what is new.
Test yourself
In the memory-augmented one-shot setup, why is the true label for a sample revealed only after the model has predicted, and what would likely go wrong if labels arrived simultaneously with their inputs throughout training?
The delay forces the model to store the input–label binding in external memory and to retrieve it on later encounters; success on a repeated or similar input becomes evidence that write and read worked. If labels always arrived with inputs, the controller could lean on short-term copying or on weight-side shortcuts that never practise genuine storage of a novel class for later use. The policy for one-shot filing would not be under pressure to form, and the external memory would not have to earn its keep as the seat of episode-specific knowledge.
Go deeper
- [1605.06065] One-shot Learning with Memory-Augmented Neural Networks · arxiv.org
- [1703.03400] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.