II · THE IDEA · ARTIFICIAL INTELLIGENCE
Multi-Task Learning with Task Balancing
▶ Listen · narrated
Train depth and semantics together and one loss often swamps the other. Letting the model learn how uncertain each task is restores balance without hand-tuned scalars.
At a glance
- Problem
- One task loss can dominate shared parameters in joint training
- Approach
- Learn a weight for each task from its homoscedastic uncertainty
- Effect
- Automatically scales each task's contribution to the total loss
- Setting
- Scene geometry and semantics trained in one network
Think of two people giving directions to the same driver at once. One shouts in metres, the other whispers in street names. If the driver only hears the loud voice, the quiet advice never shapes the route—even though both matter.
Multi-task learning is similar: one network tries to solve several jobs at once, such as estimating depth and labelling what each pixel is. Their error scores do not sit on the same scale, so one job can drown out the other when the network updates its shared parts.
Uncertainty weighting gives each job a dial the network can turn. The dial tracks how noisy or hard that job currently seems. A noisier job is told to speak more softly in the combined score; a clearer job may speak up. A penalty stops the network from turning any dial all the way to silent. Both jobs keep a real say in training, without someone guessing fixed volume knobs in advance.
Homoscedastic uncertainty weighting casts each task likelihood with its own noise parameter and forms a joint loss in which task i’s residual term is scaled by a decreasing function of that parameter (for example loss_i / (2 σ_i²) for Gaussian regression, with an analogous temperature-style form for classification). A log σ_i regulariser arises from the normalisation of the likelihood and prevents σ_i from diverging so as to null the task. The σ_i (or s_i = log σ_i², for stable optimisation) are trained jointly with network weights, yielding adaptive task coefficients without a manual grid search.
Limits follow from the assumptions. The uncertainty is task-global, not input-dependent; it absorbs scale and average noise, not per-pixel difficulty. Misspecified likelihoods (wrong noise model, heavy tails) weaken the interpretation as maximum likelihood, though the loss may still act as a useful adaptive reweighting scheme. The method balances loss magnitudes; it does not directly equalise gradient norms in parameter space, so pathological curvature or conflicting gradient directions can remain. Empirically it was shown on scene geometry and semantics sharing a trunk; transferring the same schedules and initialisations to very different task mixtures still requires care.
Look closer
Weights are not fixed scalars
In a naive multi-task setup the total loss is a sum of task losses, each multiplied by a hand-chosen coefficient. Those coefficients are brittle: a modest change can tip the shared backbone toward one head and starve another. Uncertainty weighting replaces the fixed coefficients with parameters the optimiser itself updates, so the relative influence of each task can move as training proceeds rather than staying locked to a guess made before the first step.
Uncertainty here is task-level, not pixel-level
The method uses homoscedastic uncertainty: a single scalar per task that captures how noisy or hard that whole objective is under the current model, not a varying map over space. A regression task such as depth and a classification task such as semantic labelling live on different numerical scales; the learned uncertainty absorbs that mismatch so their gradients are not compared raw. The same idea extends when more than two heads share the trunk.
A regulariser stops the weights collapsing
If the model could drive a task weight to zero without penalty, it would simply ignore the harder objective. The formulation therefore includes a term that grows as the uncertainty parameters become large, discouraging the trivial solution of switching a task off. What remains is a compromise: down-weight a noisy or poorly scaled loss, but not so far that it vanishes from the joint optimisation.
The story
Multi-task learning puts more than one prediction head on a shared network. The hope is that features useful for depth help semantics, and features useful for semantics help depth, so each task borrows signal the other would not have found alone. The practical obstacle is the joint loss. Depth errors are continuous and often large in magnitude; classification losses live on a different scale entirely. Add them without care and the steeper or larger term dictates almost every update to the shared weights. The auxiliary task is still present in the graph, yet its gradient is too small to matter.
A common remedy is to multiply each task loss by a fixed coefficient and search for coefficients that keep training stable. That search is expensive, and the best values can shift as the network improves. The uncertainty-weighting approach treats those coefficients as learnable. Each task is given a homoscedastic uncertainty parameter—an estimate of how much irreducible noise that objective carries under the model. The task loss is then scaled by a decreasing function of that uncertainty, so a task the model currently finds noisier or harder contributes less aggressively to the total. An additional regularisation term depends on the uncertainty parameters themselves and rises if they grow without bound, which prevents the optimiser from silencing a task by driving its weight to zero.
Because the uncertainty parameters are trained with the rest of the network, the balance among tasks is no longer a hyperparameter chosen once and frozen. Early in training, when one head is badly scaled relative to another, its uncertainty can rise and temper its pull on the shared trunk. Later, as losses settle, the relative weights can shift again. The method was demonstrated on scene understanding, where geometry and semantics are natural companions: both read the same image, both benefit from edges and surfaces, and both suffer if the backbone is captured by a single dominant objective.
The idea does not claim that uncertainty is the only sensible balancing signal, nor that every multi-task problem needs learned weights. It does show that when losses sit on incompatible scales, letting the model adjust their magnitudes from task-level noise estimates is a workable alternative to manual search—and that the regulariser is essential if those estimates are not to collapse into a one-task solution in disguise.
Why it mattered then
As deep networks began to serve several scene-understanding heads from one backbone, practitioners met a stubborn tuning problem: joint training often needed careful loss coefficients, and those coefficients did not transfer cleanly across datasets or architectures. Uncertainty weighting offered a principle rather than another grid search. By tying each weight to a homoscedastic noise parameter and regularising that parameter, it gave a single, differentiable account of why one task should temporarily yield to another. In geometry-and-semantics settings, that mattered because both outputs were already being asked of robots and mapping systems; a method that kept both alive without exhaustive coefficient sweeps fitted the engineering moment.
Why it matters now
Shared backbones with multiple heads remain standard—detection with depth, language with vision, several specialist losses on one large model. The same imbalance appears whenever objectives differ in scale, batch statistics, or learning speed. Learned task weights, of which uncertainty weighting is an early and still-cited form, are one family of answers alongside gradient-norm methods and dynamic sampling. Understanding the original uncertainty formulation clarifies what is being optimised: not only the task losses, but a model of how much each loss should count. That distinction still guides how people debug a multi-task run when one metric stalls while another races ahead.
The surprising detail
The balancing weights are not hand-set importance scores. They emerge from a probabilistic reading of each task as an observation corrupted by its own noise level. Larger estimated noise shrinks that task’s pull on the shared weights—yet the log-uncertainty regulariser stops the model from declaring every difficult task infinitely noisy and walking away. The machinery that looks like mere loss scaling is, in the paper’s framing, maximum likelihood under task-dependent noise.
What is disputed
The second listed source concerns exposure misclassification and cluster size, not multi-task loss weighting; the account here follows only the uncertainty-weighting multi-task work. How far homoscedastic task noise matches real error structure is an assumption of that method, not a measured fact for every domain.
Remember this
Uncertainty weighting lets the model learn how loudly each task may speak—so no single loss monopolises the shared parameters, and no task is quietly switched off.
Test yourself
In uncertainty-weighted multi-task learning, what stops the optimiser from simply driving a difficult task’s weight to zero and ignoring it?
A regularisation term on the uncertainty parameters that grows as those parameters become large. Without it, silencing a hard task would reduce the total loss; with it, the model may down-weight a noisy objective but cannot discard it for free.
Go deeper
- [1705.07115] Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics · arxiv.org
- [1910.07438] On the Interplay Between Exposure Misclassification and Informative Cluster Size · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.