II · THE IDEA · ARTIFICIAL INTELLIGENCE
Red Teaming with Language Models
▶ Listen · narrated
Manual red teaming catches some harms and misses others. When the attacker is itself a language model, the test set grows large enough to reveal patterns that a handful of people would never sample.
At a glance
- What it is
- Generating adversarial prompts at scale to elicit harmful or undesirable model behaviour
- Who attacks
- Language models (automated) or crowdsourced humans, depending on the method
- What is scored
- Whether the target's reply is harmful, often via a separate classifier
- Aim
- Surface diverse failure modes before deployment, then use them to reduce harms
Think of a locksmith who hires another locksmith to try every odd key shape they can invent, as fast as they can, while a referee watches whether the door opens when it should not. Here the first locksmith is the target language model, the second is a generator model writing prompts, and the referee is usually a classifier that flags harmful replies.
The generator proposes an input. The target answers. The referee scores the answer. If the answer is bad in the way you care about — insults, dangerous advice, memorised private strings, and so on — you keep that prompt as evidence of a weakness. You then push the generator to try something different, not just the same attack reworded, so you learn about more than one kind of failure. People can play the generator's role too; human red teamers often find different problems than the automated loop.
Nothing in this process proves the door is safe. It only shows which keys were tried and which ones worked. The value is volume and variety: far more odd keys than a small team would cut by hand, collected early enough that you can train or filter the target before release.
Automated red teaming casts adversarial evaluation as a generate–and–score loop over natural-language prompts. A generator LM, steered by zero-shot or few-shot instructions, supervised fine-tuning on prior successful attacks, or RL with rewards for target harmfulness plus prompt novelty, samples candidate inputs x. The target model produces y ~ p_target(·|x). A scoring function s(x,y) — often a separate classifier for offensiveness, or a task-specific detector for leakage or policy breach — decides whether the exchange counts as a hit. Hits are retained as an attack dataset; that dataset can drive further generator training or downstream mitigation on the target (for example rejection sampling or preference-style fine-tuning against harmful exchanges).
Two design pressures dominate. First, coverage: without an explicit diversity or novelty term, generators concentrate on high-reward modes and recycle near-duplicates, inflating hit rate while under-testing the input space. Second, measurement: s is a proxy. Classifier error, prompt sensitivity, and disagreement with human raters all bias reported success rates. Human red-teaming studies supply a complementary distribution of attacks and the practical data used to reduce harms; they also show non-monotonic scaling effects, including cases where larger targets produce more fluent harmful text on compliance even when refusal training later improves. The loop is iterative rather than conclusive: after mitigation, the attack distribution shifts, so generators and human teams must be re-run. The method improves sample efficiency of adversarial evaluation; it does not yield a soundness guarantee over all prompts.
Look closer
The attacker is also a model
In the automated setting, one language model is prompted or trained to produce candidate inputs for a second, target model. The target's replies are then judged — often by a classifier — for offensiveness, leakage of training data, or other undesirable traits. Successful attacks are kept; weak ones are discarded. The loop can run without a person writing each prompt by hand, which is what makes the volume of test cases practical.
Diversity is the hard part
A generator that repeats slight variants of the same insult only probes one corner of the failure surface. Zero-shot and few-shot prompting, supervised fine-tuning on earlier attacks, and reinforcement learning that rewards both harmfulness of the target's reply and novelty of the prompt have all been used to push the attacker toward a wider set of cases. Without an explicit diversity pressure, the method collapses into a narrow band of attacks that look serious in aggregate counts but leave most behaviours untested.
Humans and scale still matter
Parallel work with crowdsourced human red teamers shows that people find different classes of harm, and that attack success changes with the size and training of the target. Larger models can be easier to goad into fluent harmful text in some settings, yet better able to refuse once they have been trained against red-team data. Automated generation and human red teaming are complementary samples of the same underlying risk surface, not substitutes.
The story
Red teaming, in this setting, means deliberately trying to make a language model do something its designers do not want: produce offensive content, leak memorised training data, exhibit distributional bias, assist with harmful instructions, and so on. The traditional way to do this is to hire people to invent tricky prompts. That works, but it is slow, expensive, and shaped by whatever the red teamers happen to think of.
The automated approach treats attack generation as a language-modelling problem in its own right. A generator model proposes prompts; the target model answers them; a classifier or scoring rule decides whether the answer counts as a failure. Prompts that succeed become evidence of a real weakness. The generator can be steered with ordinary prompting — instructions and a few examples of the kind of attack wanted — or trained more heavily. Supervised fine-tuning on previously successful attacks teaches it what has worked. Reinforcement learning can go further, rewarding prompts that both elicit a high harm score from the target and differ from attacks already found, so the search does not stall on near-duplicates.
What emerges is not a single jailbreak but a distribution of failures. Offensive language, unsolicited advice on dangerous topics, and leakage of memorised strings are among the categories reported. The same pipeline also shows how uneven the surface is: some harms are easy to elicit with short, blunt prompts; others need longer setups or particular framings. Because the generator can keep sampling, rare failure modes that a small human team would be unlikely to hit in a fixed budget of hours can still appear in the collected set.
A second line of work pairs this idea with large-scale human red teaming and with downstream mitigation. Crowdsourced red teamers produce attack data that is then used to refine the target — for example by rejecting fine-tuning or other preference-style training against the harmful exchanges. Scaling behaviour is not uniform across harm types. In some evaluations, larger models produce more convincing harmful text when they comply, which makes successful attacks more serious even when refusal rates improve. After training on red-team data, models can refuse more often, but the residual failures shift rather than vanish: new prompts, or prompts outside the distribution of the red-team set, continue to land.
The practical picture is therefore a loop rather than a one-off test. Generate diverse attacks (by model, by humans, or both); score the target's behaviour; keep the failures; train or filter to reduce them; repeat. The method does not prove a model is safe. It only increases the chance that particular classes of bad behaviour are seen before deployment, under the attack distribution the red team actually sampled. That limitation is structural: the generator and the human team both have blind spots, and the classifier that labels harm can be wrong in both directions. Even so, once language models are capable enough to propose fluent, varied prompts, using them as red teamers becomes one of the few ways to put meaningful volume behind adversarial evaluation.
Why it mattered then
As language models moved from research demos into products that millions of people could query, failure modes stopped being anecdotal. A handful of embarrassing screenshots was no longer an adequate risk picture. Teams needed a way to sample adversarial inputs systematically before release, not only after users found them. Automated red teaming arrived in that gap: it borrowed the models' own generative strength to stress-test their behaviour at a scale manual review could not reach, and it produced reusable datasets of failures for training-time mitigation. Human red teaming, run in parallel at large scale, supplied a complementary sample and the data used to reduce harms in subsequent model versions. Together they marked a shift from ad hoc probing to red teaming as a repeatable evaluation and training ingredient.
Why it matters now
Deployed models still face an open-ended user population and an open-ended space of prompts. Manual test suites freeze a small slice of that space; attackers and ordinary users do not. Automated red teaming remains one of the main tools for expanding the slice before and after release, and for regenerating attacks after each safety intervention so that regressions and shifted failures are visible. The same loop also forces uncomfortable measurement questions that still matter in practice: how to score harm without a perfect classifier, how to keep attack diversity from collapsing, and how to avoid mistaking “fewer hits under our generator” for general safety. Those questions sit inside current evaluation practice whenever a lab claims a model has been adversarially tested.
The surprising detail
A consistent practical finding is that simply making the target larger does not steadily make red teaming harder. In some setups, scale makes harmful completions more fluent and therefore more useful to an attacker when the model complies, even as other training stages improve refusal. Another sharp result is how quickly an unregularised attack generator collapses onto a narrow family of prompts: without an explicit novelty or diversity term, volume goes up while coverage does not. The method's power and its main failure mode are the same fact — it optimises for what you measure.
What is disputed
Harm classifiers disagree with human judgements and with each other, especially on borderline or context-dependent cases. Reported attack success rates are therefore tied to a particular scoring rule and should not be read as absolute counts of harm. Different papers also operationalise “diversity” and “harm” differently, so numerical results are not directly interchangeable across studies.
Remember this
Red teaming with language models samples failures at scale; it does not exhaust them. Diversity of attacks, and honesty about the scorer, matter as much as raw hit rate.
Test yourself
An automated red-team run reports a high rate of successful attacks, but nearly all hits are slight rephrasings of the same request. A second run, after a diversity reward is added, reports a lower hit rate spread across many distinct harm types. Which run is more informative for deciding what to fix before release, and why?
The second. A high hit rate on near-duplicate prompts mostly shows that one narrow failure is easy to elicit, and it overstates how thoroughly the model was probed. The more diverse set, even with fewer successes, maps more of the behaviour surface and gives training or filtering something broader to work against. Hit rate without diversity is a weak safety signal.
Go deeper
- [2202.03286] Red Teaming Language Models with Language Models · arxiv.org
- [2209.07858] Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.