II · THE IDEA · ARTIFICIAL INTELLIGENCE
Multi-Agent Deep Reinforcement Learning with Centralized Critics
▶ Listen · narrated
Independent learners in a shared world keep rewriting each other's outcomes. Training with a joint critic, then executing alone, is one way to keep policies local while still assigning credit with global context.
At a glance
- Core idea
- Centralised training, decentralised execution via per-agent critics
- Method
- Multi-agent deep deterministic policy gradient (MADDPG)
- Critic input
- All agents' observations and actions during training
- Actor input
- Only each agent's local observation at execution
- Environments
- Cooperative, competitive, and mixed continuous-action tasks
Think of a football side where each player may only look through their own eyes on match day, yet during the week a coach watches everyone at once and tells each player which decisions helped the team. On the pitch the coach is gone; only the practised habits remain.
Multi-agent deep reinforcement learning faces the same split. If every agent learns in isolation, teammates and opponents keep changing, so the same local choice no longer leads to the same result. A centralised critic plays the coach during training: it is allowed to see all agents' observations and actions when scoring how good a decision was. Each agent's actor—the policy that picks actions—still only sees its own input, so after training the agents can run without sharing senses. The method extends deterministic policy gradients to this multi-agent pattern and works with cooperative, competitive, or mixed rewards.
Single-agent actor-critic methods such as A3C stabilise policy gradients with parallel learners, but they assume environmental stationarity. In multi-agent learning that assumption fails: other agents' evolving policies make P(s'|s,a_i) non-stationary from agent i's perspective, which destabilises both replay-based and on-policy updates.
MADDPG applies centralised training with decentralised execution. For N agents, each has an actor μ_i(o_i) and a critic Q_i(x, a_1, …, a_N), where x collects joint observations (or relevant state). Because the critic conditions on the full action tuple, the regression target y = r_i + γ Q_i(x', a_1', …, a_N')|_{a_j'=μ_j(o_j')} does not treat other policies as hidden non-stationarity—their actions are arguments. Actors are updated with the deterministic policy gradient using only ∂μ_i/∂θ_i and the critic's derivative with respect to a_i; at execution, only μ_i(o_i) is required.
Separate Q_i allow heterogeneous rewards, covering cooperative, competitive, and mixed continuous-action environments. Limitations include critic input dimensionality growing with N, reliance on access to other agents' actions at train time, and no guarantee of equilibrium selection when incentives conflict.
Look closer
Why independent learners stall
From any single agent's view, other learning agents make the transition dynamics look non-stationary: the same action in the same local state can yield different returns simply because teammates or opponents changed their policies. Standard single-agent actor-critic and Q-learning assume a stationary environment, so their learning targets keep shifting as the other policies update. The multi-agent actor-critic work treats that non-stationarity as the central obstacle rather than a side effect.
What the centralised critic actually sees
During training, each agent's critic is allowed extra information that will not be available at execution: the observations and actions of the other agents. Conditioning the critic on the full joint action makes the environment stationary from the critic's perspective even while other policies are changing, because those policies' current actions are explicit inputs. The actor for each agent is still updated only from its own observation, so the learned policy remains locally executable.
Deterministic policies and mixed incentives
The method extends deep deterministic policy gradient to multiple agents, fitting continuous action spaces. Separate per-agent critics and actors allow different reward structures at once: agents may share a reward, oppose one another, or sit in mixed settings where some cooperate and others compete. The paper evaluates these regimes rather than assuming a single fully cooperative team.
The story
Deep reinforcement learning had already shown that an actor and a critic can be trained together at scale. Asynchronous advantage actor-critic, for example, used parallel actor-learners to stabilise policy-gradient training without relying on experience replay. That line of work still largely addressed a single agent facing a fixed environment.
Multi-agent settings break that assumption. As soon as several agents learn at once, each agent's experience is generated by a world whose effective dynamics depend on the current policies of the others. A replay buffer full of old joint behaviour can become misleading; even on-policy updates face a moving target because the same local observation no longer predicts the same return. Credit assignment becomes tangled: an agent cannot easily tell whether a success came from its own action or from a teammate's change of behaviour.
The multi-agent actor-critic approach associated with MADDPG keeps the convenient deployment model people actually want—each agent acts from its own observations at test time—while changing what is allowed during training. Each agent retains its own actor, a policy that maps local observation to action. Each agent also has a critic used only in training. That critic is permitted to condition on the observations and actions of all agents, not merely its own. Once the joint action is an input, the critic's learning target no longer has to treat other agents as an invisible, shifting part of the environment; their current behaviour is observed directly.
Because the critics are centralised only for the update, the actors can still be trained with gradients that depend solely on local information at execution. Policies therefore remain decentralised: no agent needs to broadcast its observations or actions when the system is deployed. The method builds on deterministic policy gradients, which suits continuous control, and equips each agent with its own pair of networks so that rewards need not be identical. Cooperative teams, competitive adversaries, and mixed populations can be expressed in the same framework simply by how rewards are assigned.
The practical effect is a cleaner separation between the information needed to assign credit and the information needed to act. Training uses a privileged joint view to stabilise the critic; execution discards that view and keeps only the local actors. That split is the editorial point of the work: complex decentralised policies become easier to learn when credit assignment is allowed to look over everyone's shoulder—then forget what it saw.
Why it mattered then
By the mid-2010s, deep actor-critic methods had made single-agent continuous control far more practical, yet multi-agent benchmarks still exposed a structural gap. Independent learners, each running a standard algorithm, interfered with one another through the shared dynamics; fully centralised controllers avoided that interference but gave up the decentralised execution that many applications require. The MADDPG line offered a middle path that matched how people already wanted to deploy multi-agent systems: train with extra joint information, ship only local policies. It also spoke to mixed incentive settings rather than only pure cooperation, which widened the set of environments where the same recipe could be tried.
Why it matters now
Decentralised execution with centralised training remains a default pattern whenever several learned policies must coordinate or compete without a permanent central controller—robot teams, traffic agents, games, and simulated economies among them. The underlying tension has not vanished: non-stationarity still appears whenever co-learners update, and credit assignment is still harder when rewards depend on joint behaviour. Later algorithms refine critics, communication, and credit mechanisms, but they often still rely on the same permission structure this work made explicit—use global context while learning, act locally when done.
The surprising detail
The critic is allowed to see other agents' actions during training even in competitive settings, where those agents are opponents. The centralisation is a training-time privilege for value estimation, not a requirement that teammates trust one another or share a reward. The same joint-action conditioning that steadies cooperative credit assignment also applies when the other policies are adversarial; what changes is only the reward each critic is asked to predict.
What is disputed
The supplied papers establish the non-stationarity argument and the centralised-critic remedy for continuous multi-agent tasks; they do not claim this is the only stable approach, nor that centralised critics remove all coordination failure. Empirical gains depend on task structure, reward design, and how much joint information the critic actually receives.
Remember this
Centralise the critic to steady credit assignment while agents learn; keep actors local so policies still run alone.
Test yourself
An engineer trains two agents with centralised critics that condition on both agents' actions, then deploys only the actors. At deployment one agent cannot observe the other. Why might learning still have succeeded where two fully independent actor-critic learners failed?
During training each critic saw the joint action, so other-agent policy changes entered as observed inputs rather than as unexplained shifts in the environment; that kept the critic's regression target nearer to stationary. The actors only ever needed local observations, so removing the extra inputs at deployment does not break the learned policies. Independent learners never had that joint view, so each treated the other's learning as non-stationarity and their value targets kept drifting.
Go deeper
- [1602.01783] Asynchronous Methods for Deep Reinforcement Learning · arxiv.org
- [1706.02275] Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.