II · THE IDEA · ARTIFICIAL INTELLIGENCE
Multi-Task Reinforcement Learning with Shared Representations
▶ Listen · narrated
Separate policies for every environment waste experience. A joint representation lets structure discovered in one task shorten the climb in the next.
At a glance
- Core idea
- Related tasks share a feature space rather than learning wholly separate ones
- Payoff
- Faster adaptation when a new task reuses structure already learned
- Mechanism
- Common layers or encoders, with task-specific heads on top
- Main risk
- Negative transfer if tasks pull the shared space in opposing ways
Think of several apprentices sharing one notebook of sketches. Each apprentice draws from different jobs, but they all add to the same pages. Over time the notebook fills with useful outlines—doorways, edges, rough maps—that any of them can reuse. When a new apprentice arrives for a similar job, they do not start with blank paper; they borrow the notebook and only learn the last details of their own task.
That is the idea behind a shared representation. One common part of the model sees experience from every related task. Separate small parts on top handle what is unique to each task. If the tasks truly resemble one another, the common part becomes a head start. If they do not, the shared notebook fills with mixed advice and can slow everyone down.
Multi-task learning with a shared trunk parameterises a feature map f_θ used by task-specific heads h_i. For reinforcement learning those heads are typically policies π_i or value functions V_i (or both). A joint objective aggregates task losses—or policy-gradient terms—often with scalar weights λ_i, and minibatches that interleave environments or cycle on a fixed schedule.
Gradients ∂L_i/∂θ from every active head update the same θ. Features that reduce loss across several tasks are reinforced; features useful to only one task survive only if that task is sampled and weighted enough. Adaptation to a new related task k then freezes or lightly fine-tunes θ and trains h_k, which is where the sample-efficiency claim is supposed to appear.
Failure modes are concrete: gradient dominance by high-variance or dense-reward tasks; representation collapse toward the majority domain; and negative transfer when tasks need incompatible invariances. Mitigations include loss normalisation, gradient projection or surgery, partial sharing (task-private upper layers), and routing or mixture-of-experts variants when a single flat trunk is too tight a bottleneck.
The same trunk-and-heads pattern is used in multi-task deep networks for natural language understanding, where a shared encoder is trained across several supervised objectives with lightweight task heads. The parallel is architectural, not a claim that language and control share hyperparameters or scaling laws.
Look closer
What is actually shared
The early part of the network is asked to serve more than one objective at once. Gradients from each task update the same trunk weights, so the representation is pressured to keep features that help across the set and to drop those that only ever matter for a single narrow case. Task-specific heads then read from that common space without forcing every decision through identical final weights.
Where sample efficiency appears
When a newly encountered but related setting arrives, the shared layers may already carry useful structure—recurring motifs in the input, coarse regularities, patterns of reward. Only the head, or a light adaptation of the trunk, then needs substantial new data. The saving holds only when the new task genuinely lives in the same family; otherwise old features become a bias that must be unlearned.
The tension you can watch in training
On a joint loss surface one task can improve while another stalls or slips. That pattern is the practical signature of representation conflict. Balancing sampling rates, loss weights, or gradient norms is how the shared space is kept from collapsing toward the loudest task. The pipeline diagram is simple; the mixing schedule is where the method lives or dies.
The story
Multi-task learning with shared representations starts from a plain observation: many problems are not strangers to one another. They reuse structure in the input, similar notions of progress, or overlapping skills. If each task trains a private encoder from scratch, every run pays the full cost of discovering that overlap again. Sharing the encoder—or a substantial trunk of it—lets experience gathered under one objective shape features that another objective can read.
A common arrangement is straightforward. A backbone maps observations into a feature space. From those features, separate heads produce the outputs each task needs—policies or value estimates in a reinforcement-learning setting, classification or scoring heads in a language-understanding setting. During training, batches may interleave tasks or cycle through them on a schedule. Gradients flow back into the backbone from whichever head was active, so the representation is co-authored by the whole family of tasks.
What the backbone keeps is not guaranteed to be elegant. It keeps what survives the joint pressure of the losses. Features that help two or more tasks tend to stabilise; features that help only one may shrink unless that task is sampled often enough or weighted strongly enough. This is why multi-task training is not automatic transfer. It is a negotiation, and the terms of the negotiation are set by how tasks are mixed and how conflicts are damped.
Work on multi-task deep networks for natural language understanding has made the same structural bet in a different domain: a shared trunk trained on several related objectives, with lightweight task-specific layers on top. The editorial angle here carries that bet into reinforcement learning. Related environments can, in principle, co-train a feature space that a newcomer reuses, so that competent behaviour in the new setting arrives with fewer fresh samples than a cold start would need.
The promise is conditional. Sharing helps when tasks truly agree on what the representation should preserve. When they disagree—different sensors, opposing reward pressures, mismatched timescales—the shared trunk can become a compromise that serves no task well. Practitioners then fall back on partial sharing, gradient surgery, or simply splitting the conflicting tasks apart. The method earns its keep only where the family resemblance among tasks is real, not assumed.
Why it mattered then
As soon as learners were asked to handle suites of related problems rather than a single fixed environment, the cost of training a private representation for each member of the suite became obvious. Shared trunks were an engineering response to that cost: one set of features, many heads, and a hope that joint training would regularise the representation instead of merely averaging incompatible demands. The idea mattered because sample collection in interactive settings is slow and expensive; anything that reused structure across tasks was worth trying.
Why it matters now
Agents are still expected to meet families of environments—varied instructions, altered layouts, shifted reward definitions—rather than one frozen task. Shared representations remain one of the plainest tools for spending less new experience on each variant. The same design pattern appears outside reinforcement learning, including in multi-task language models that share a trunk across several understanding objectives, which is a reminder that the pressure to amortise representation learning is not unique to control.
The surprising detail
The hard part is rarely drawing the shared-trunk diagram. It is noticing when joint training is quietly harming a member of the suite. A task can lose accuracy while the average score still rises, because the backbone has drifted toward its neighbours. Without per-task curves, negative transfer hides inside a flattering aggregate.
What is disputed
The sources supplied for this lesson include a multi-task deep network paper for natural language understanding and an unrelated polarimetry study. Claims here about reinforcement-learning sample efficiency are therefore reasoned from the shared-representation pattern, not from numbered results in those sources. Where tasks diverge, the size of any benefit—or harm—must be measured per suite; it is not guaranteed by the architecture alone.
Remember this
A shared representation buys faster adaptation only when the tasks truly agree on what is worth keeping.
Test yourself
Two navigation tasks share an encoder. One is sparse-reward and long-horizon; the other is dense-reward and short. After joint training, the sparse task is worse than when trained alone. What is the most plausible representation-level explanation, and what change would you try first?
The dense short-horizon task likely dominates the gradient signal, so the shared features specialise in local, immediately rewarded cues and under-serve long-horizon structure the sparse task needs. That is negative transfer through the trunk. First try re-balancing: up-weight the sparse task’s loss, oversample it, or reduce the dense task’s learning rate into the shared layers. If conflict remains, share fewer layers and keep higher layers task-private.
Go deeper
- [1709.06061] Optical Polarimetry of KIC 8462852 in May-August 2017 · arxiv.org
- [1901.11504] Multi-Task Deep Neural Networks for Natural Language Understanding · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.