II · THE IDEA · ARTIFICIAL INTELLIGENCE
Red Teaming for Language Model Safety
▶ Listen · narrated
A model can pass a fixed safety suite and still fail the moment a determined person tries to make it misbehave. Red teaming is that determined person, organised.
At a glance
- What it is
- Systematic adversarial probing to elicit harmful model behaviour
- Human role
- Testers craft prompts that push models toward unsafe outputs
- LM-assisted
- Language models can generate diverse attack prompts at scale
- Training use
- Collected failures can drive harmlessness training and AI feedback
- Limit of suites
- Fixed automated benchmarks miss novel or creative attack paths
Think of a locksmith hired to break into a building before the thieves try. Red teaming does that for language models: people deliberately send prompts meant to draw out unsafe answers — instructions for wrongdoing, personal data leaks, cruel or disallowed content — and record what works.
Ordinary safety tests are more like a fixed checklist of doors. Useful, but a clever person will try windows the checklist never named. Red teaming is the clever person.
Researchers also found that another language model can propose many of these attack prompts automatically, so the search can run at larger scale than a small human team alone. When an attack succeeds, the example is not only a bug report. It can become training material: the model is taught to refuse, to answer more carefully, or — in constitutional AI — to critique and revise its own reply against a written list of principles before answering.
The point is not a single dramatic failure. It is a repeating cycle: probe, label, train or adjust, probe again with new attacks.
Red teaming for LM safety is systematic adversarial elicitation of policy-violating outputs. Human red teamers craft single-turn and multi-turn prompts that target refusal failures, over-refusal edge cases, dual-use assistance, and related harms. Successful threads become labelled data for supervised refusal training, preference modelling, or offline evaluation slices that frozen benchmarks omit.
Perez et al. (arXiv:2202.03286) automate attack proposal with LMs. A red-team model generates candidate prompts (zero-shot, few-shot, or learning-augmented); the target completes them; a classifier or human judge scores harm. The objective is coverage and diversity across harm taxonomies, not a single jailbreak string. Failure modes found this way include toxic content, sensitive-information leakage patterns, and assistance that violates stated usage policies.
Bai et al. (arXiv:2212.08073) describe Constitutional AI: critiques and revisions conditioned on a written constitution, then reinforcement learning from AI feedback for harmlessness, aiming to reduce dependence on human preference labels for every comparison. Red teaming remains complementary — an outer-loop pressure test of whether constitutional training and related alignment stages hold under adaptive prompts.
Limitations: harm labels are policy-relative; automated harm scorers miss context and false positives/negatives; LM attackers can mode-collapse into repetitive strategies without diversity pressure; and closing one attack family does not prove robustness to the next. Engineering practice is iterative: attack generation → judgement → data curation → train/revise → regression plus fresh red teams.
Look closer
The attack is the dataset
Red teaming is not a one-off audit. Each successful attack — a prompt that draws a harmful, evasive, or policy-breaking reply — becomes labelled evidence. In later training stages those examples can teach a model what to refuse, how to refuse, and when a revision is safer than the first draft. The quality of safety training is therefore bounded by the imagination and persistence of the attacks collected.
Models attacking models
Perez and colleagues showed that a language model can itself propose the adversarial prompts. Zero-shot and few-shot generation, then stronger learning-based variants, produce large volumes of candidate attacks against a target model. Diversity matters as much as volume: a narrow generator rediscovers the same insult patterns, while a broader one reaches data leakage, assistance with wrongdoing, and other failure families that a small human team may under-sample.
Principles in the loop
Constitutional AI uses a written list of principles — a constitution — so that a model can critique and revise its own replies toward harmlessness, then provide AI feedback for reinforcement learning. Red teaming still sits in the surrounding practice: adversarial human probes remain a way to pressure-test whether the constitution and the training actually hold when someone is trying to break them.
The story
Safety evaluation for language models has an awkward shape. A fixed test suite can check known refusal categories, toxicity classifiers, and scripted edge cases. It cannot easily anticipate the next creative phrasing, the multi-turn bait, or the polite request that conceals a harmful goal. Red teaming answers that gap by putting adversarial effort on the input side: people (and, increasingly, models) try to make the system fail on purpose.
In the human form of the practice, testers treat the model as an opponent. They rephrase, escalate, role-play, and search for contexts where helpfulness collides with a safety policy. What they surface is often not a single bug but a family of behaviours — over-compliance, partial leaks of sensitive patterns, instructions that should have been refused, or answers that are technically non-actionable yet still harmful in aggregate. Automated benchmarks tend to lag these discoveries because the benchmark authors have to already know what to look for.
Perez et al. asked whether language models could scale the generation of such tests. Their approach is straightforward in outline. A red-teaming model proposes prompts intended to elicit bad behaviour; the target model answers; scorers or human judges decide whether the exchange counts as a failure. Generation can be zero-shot or few-shot, or improved with learning so that the attacker becomes better at finding weak spots. The practical gain is coverage. A small group of human red teamers will eventually repeat itself; a generator that explores many phrasings can keep producing fresh candidates across categories of harm.
Those candidates only matter if they feed back into training. Failures become examples for refusal behaviour, preference data, or critique-and-revision loops. Constitutional AI, described by Bai and colleagues, is one such loop structured around a written constitution: the model critiques its own draft against stated principles, revises toward a less harmful reply, and later supplies AI feedback used in reinforcement learning for harmlessness. Red teaming and constitutional training are complementary rather than interchangeable. One pressure-tests the system from outside; the other tries to instil a stable internal standard so that ordinary prompts, not only adversarial ones, land in a safer region of behaviour.
None of this removes the need for human judgement. What counts as harm is partly policy, partly context, and partly disputed. A prompt that looks like a clear attack in one setting may be a legitimate research or fiction request in another. Red teaming therefore tends to be iterative and organisational as much as technical: define the policies, probe against them, label carefully, retrain or revise, and probe again — including with attacks the previous round never saw.
Why it mattered then
When these papers appeared, language models were moving from research demos into products that millions of people could prompt freely. Standard evaluation suites could report progress on known tasks, yet they systematically under-sampled creative misuse. Human red teaming made the failure distribution visible in a way leaderboards did not. LM-based red teaming addressed a second problem of the moment: human attack generation does not scale cheaply, so the attack surface grew faster than the labour available to map it. Constitutional AI addressed a third: even after harmful outputs were collected, relying only on human preference labels for harmlessness was expensive and slow. Using a written constitution plus model self-critique offered a path to more harmlessness training signal without a matching increase in human annotation for every comparison.
Why it matters now
Deployed models still face users who will push them — curious, malicious, or merely unlucky in phrasing. Fixed safety benchmarks remain necessary but insufficient; they measure what the benchmark author already imagined. Red teaming continues to be how organisations discover the remainder. The LM-assisted variant matters more as models multiply and release cycles shorten, because coverage has to keep pace with new capabilities and new tool integrations. Constitutions and AI feedback remain one concrete design pattern for turning those discoveries into training pressure rather than into a static list of blocked strings. The practice also sets expectations: a model that has only been checked against a frozen suite has not been shown safe against a determined adversary.
The surprising detail
A language model can be an effective red teamer against another model — including when the attacker is not hand-tuned on a catalogue of known exploits. Zero-shot and few-shot prompt generation already surfaces diverse failure modes; learning-based attackers can improve further. That means the same general technology that produces helpful assistants also scales the search for ways they break, which tightens the loop between capability and evaluation in a way earlier software testing rarely had to confront.
What is disputed
What counts as a successful attack depends on policy definitions of harm, which differ across organisations and jurisdictions. Reported failure rates are therefore not directly comparable without the labelling rubric. Evidence that LM-generated attacks match the creativity of skilled human red teamers is encouraging in aggregate but uneven across harm categories; some subtle social manipulations remain easier for people.
Remember this
Red teaming finds the failures fixed benchmarks miss; those failures only improve safety if they are labelled and fed back into training.
Test yourself
A model scores well on a frozen safety benchmark, yet a week of human red teaming still produces many policy-breaking dialogues. Give two distinct reasons this can happen, and one reason simply collecting those dialogues is not enough to fix the model.
First, the benchmark only tests behaviours its authors already encoded; creative phrasing, multi-turn setups, and novel goal-hiding will fall outside it. Second, benchmarks are typically static and narrow in distribution, while human adversaries adapt to whatever refusals they just saw. Collecting dialogues is not enough because without labelling, policy decisions, and a training or product response — refusal tuning, critique-and-revision, filtering, or constitutional feedback — the failures remain a report, not a change in model behaviour.
Go deeper
- [2202.03286] Red Teaming Language Models with Language Models · arxiv.org
- [2212.08073] Constitutional AI: Harmlessness from AI Feedback · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.