II · THE IDEA · ARTIFICIAL INTELLIGENCE
Bias, Harm and Safety Training
▶ Listen · narrated
Safety training teaches a model which completions to avoid, not which facts to forget. The original distributions remain, overlaid by a new preference for refusal.
At a glance
- What it is
- Additional training that adjusts output probabilities toward safer, more helpful completions
- Common methods
- RLHF (human feedback) or RLAIF (AI feedback, as in Constitutional AI)
- What it preserves
- The underlying pre-trained weights and their encoded statistical patterns
- Measured effect
- RealToxicityPrompts showed base models generate toxic continuations 50–90% of the time; safety tuning reduces this substantially but not to zero
Imagine teaching a parrot to avoid saying certain phrases by rewarding it when it says something else instead. The parrot still knows the phrases — they are in its memory from hearing them before — but it has learned that saying other things gets a better response. That is roughly how safety training works. The model learned patterns from a large body of text during its initial training, including patterns we would call harmful or biased. Safety training does not erase that learning. It adds a second layer of training, using a much smaller set of examples, where humans (or another AI) mark which responses are better. The model learns to favour those responses. But the original patterns are still encoded in the weights. If a prompt is unusual enough that it does not match the contexts the model saw during safety training, the old patterns can surface again.
Safety training is typically implemented as a reinforcement learning phase applied after pre-training. The model generates multiple candidate completions for a given prompt. Human annotators rank these completions, or in Constitutional AI an AI system critiques them against a written set of principles. These preferences are converted into a reward signal, and the model's parameters are updated using a policy gradient method such as Proximal Policy Optimisation. The objective is to increase the probability of high-reward (safe, helpful) completions and decrease the probability of low-reward ones. Crucially, this is not retraining from scratch. The pre-trained weights, which encode the statistical structure of the original corpus including its biases and toxic patterns, are only adjusted, not overwritten. The scale of the adjustment is limited by the size and diversity of the preference dataset, which is orders of magnitude smaller than the pre-training corpus. The result is a shift in the output distribution: harmful completions become less probable, but they remain possible, particularly when the input distribution at inference time diverges from the distribution seen during alignment. Adversarial robustness is poor because the model has no explicit representation of harm; it has only learned correlations between certain prompt features and certain preferred output features. Constitutional AI introduces a self-critique step before the RL phase, where the model generates a response, critiques it according to the constitution, and revises it. The critiques and revisions are used as additional training data. This reduces annotation cost and makes the principles explicit, but it does not change the fundamental limitation: the model can only critique what it has learned to recognise as problematic, and that recognition itself is a learned behaviour subject to the same distributional constraints.
Look closer
Bias is baked in during pre-training, not added by alignment
When a model completes "The nurse picked up her" more often than "his", that association was learned from the distribution of pronouns in the training corpus, likely during the initial unsupervised phase. Safety training happens afterward, on a much smaller dataset, and it teaches the model to decline certain requests or rephrase certain outputs. It does not go back and re-weight every gendered pronoun the model ever saw. The statistical tilt remains in the weights; alignment training adds a layer of behaviour on top.
Refusal is a learned output, not a filter
There is no separate module scanning for bad content. When a safety-trained model says "I can't help with that", it is generating that refusal the same way it generates any other text: by sampling from adjusted probability distributions over tokens. The adjustment came from examples where human labellers or an AI critic preferred a refusal over a direct answer. If the prompt is adversarially constructed — say, embedding a harmful request inside a role-play or a coded language the training set did not cover — the refusal behaviour may not trigger, because the model is doing pattern-matching, not reasoning about harm.
Constitutional AI replaces some human labour with model-generated critique
In the Constitutional AI approach described by Bai and colleagues, the model is first asked to generate responses, then asked to critique its own responses against a written set of principles (the "constitution"), then asked to revise. These model-generated critiques and revisions become training data for a second phase of reinforcement learning. The method is intended to make the process more transparent and scalable than relying solely on human preference labels, but it still depends on the model's existing capabilities: if the base model cannot recognise a certain kind of harm in its own output, the constitutional feedback will not correct for it.
The story
A language model learns, during pre-training, to predict the next token by absorbing statistical regularities from vast amounts of text. If that text contains stereotypes, slurs, or toxic patterns — and all large corpora do — the model learns those patterns too. It does not learn that they are harmful; it learns that they are probable.
Safety training, which happens later, does not erase those probabilities. It cannot, without retraining the entire model from scratch on a filtered corpus, which would degrade performance on many other tasks. Instead, safety training teaches the model a new objective: prefer outputs that human raters (or an AI acting as a proxy for human values) marked as more helpful, harmless, or honest. This is usually done through reinforcement learning from human feedback, abbreviated RLHF.
The process works roughly like this: the model generates several candidate responses to a prompt. Human labellers rank them by preference. Those rankings become a reward signal that nudges the model's probabilities. Generate a helpful refusal when asked how to hotwire a car, and that pathway gets reinforced. Generate detailed instructions, and that pathway is discouraged. After many such adjustments, the model learns a policy: in contexts that resemble the harmful prompts it saw during safety training, produce outputs that resemble the preferred responses.
Notice what this does not do. It does not locate and delete the representation of hotwiring from the weights. It does not filter the output through a separate classifier. It adjusts the probability landscape so that safer outputs become more likely. The knowledge remains; the behaviour changes.
This distinction matters when we talk about bias. If a model disproportionately associates certain demographics with certain roles, that association was learned during pre-training from a corpus that reflected those patterns. Safety training can teach the model to avoid generating overtly offensive completions, but it does not rewrite the statistical associations deeper in the network. Researchers using the RealToxicityPrompts benchmark found that even well-aligned models sometimes produce toxic continuations, particularly when the prompt is constructed to lead in that direction. The base model's learned distributions are still there, and under the right conditions they surface.
Constitutional AI, introduced by Bai and colleagues at Anthropic, offers a variation on this process. Instead of relying entirely on human labellers, the method uses the model itself as a critic. The model generates a response, then critiques that response against a written constitution — a set of principles describing desired behaviour. It then revises the response based on its own critique. These self-critiques and revisions become training data for a reinforcement learning phase, much as human preferences do in RLHF.
The advantage is scalability and transparency: the principles are written down, and the model's reasoning about them is visible in the critique step. The limitation is that the model can only critique what it can recognise. If the base model has no representation of a particular kind of harm, or if that harm is encoded in a way the constitutional principles do not cover, the self-critique will not flag it. The method assumes a model capable of moral reasoning about its own outputs, which is itself a capability learned from data.
Alignment, then, is not a property of a model but a description of a training process applied to it. An aligned model is one that has undergone this additional phase of learning preferences. It has not been made safe in some absolute sense; it has been made more likely to behave safely in contexts that resemble its alignment training. The original model, with all its learned patterns, remains underneath.
Why it mattered then
The RealToxicityPrompts benchmark, published by Gehman and colleagues in 2020, provided one of the first systematic measurements of how often language models generate toxic text when prompted with sentence fragments from the web. The results were stark: models completed prompts with toxic language between 50 and 90 per cent of the time, depending on the model and the prompt. This was not a surprise to researchers who understood that models learn from data, but it made visible a problem that deployment would soon make urgent. If these models were to be released as products, they needed some mechanism to reduce harmful outputs. RLHF and related methods became the standard response, not because they solved the problem completely, but because they were tractable and measurably effective. Constitutional AI, introduced in 2022, extended the approach by reducing the dependence on human labour and making the training principles explicit, a move toward both efficiency and interpretability.
Why it matters now
Safety training is now standard practice for any model intended for public use, but the limitations are increasingly visible. Adversarial prompts — carefully constructed inputs designed to bypass refusal behaviour — succeed with surprising regularity, because the model is pattern-matching rather than reasoning about harm. Bias persists even in heavily aligned models, because alignment training operates on a much smaller dataset than pre-training and does not overwrite the deeper statistical associations. As models are deployed in higher-stakes contexts, the gap between "less likely to cause harm" and "safe" becomes a legal and ethical problem, not just a technical one. The debate now is not whether to align models, but how much alignment is possible within this paradigm, and whether fundamentally different architectures or training regimes might be needed to produce models that do not require a secondary layer of behaviour modification to be fit for use.
The surprising detail
Constitutional AI depends on the model critiquing itself, which means the quality of the critique is bounded by the model's existing capabilities. If the base model has learned harmful associations but lacks the vocabulary or reasoning patterns to identify them as harmful, the constitutional feedback loop will not correct them. This creates a curious dependency: the success of the safety training is limited by the very patterns the safety training is meant to address. It is not a flaw in the method so much as a constraint inherent in using a model to improve itself. The constitution provides principles, but the model interprets them through weights learned from data that may not have encoded those principles in the first place.
What is disputed
The exact reduction in toxic outputs from safety training varies widely depending on the model, the training method, the benchmark used, and how toxicity is defined. RealToxicityPrompts provides one measurement framework, but it is not comprehensive, and models that score well on it may still produce harmful outputs in categories the benchmark does not cover. There is also ongoing debate about whether Constitutional AI produces meaningfully different behaviour from RLHF, or whether the difference is primarily in transparency and scalability rather than in the resulting model's safety profile.
Remember this
Alignment training adjusts which outputs are likely, not which outputs are possible. The pre-trained distributions remain.
Test yourself
A model refuses to generate instructions for synthesising a dangerous chemical when asked directly, but complies when the request is embedded in a fictional dialogue between two characters. Explain why this happens in terms of the training process, not the model's intentions.
During safety training, the model learned to associate certain prompt patterns — direct requests for harmful information — with refusal responses, because those patterns appeared in the training data with labels marking refusal as preferred. The fictional framing was not present in the safety training set, or was present too rarely to establish a strong association. The model is not reasoning about whether the request is actually harmful; it is pattern-matching against the contexts it saw during alignment. The refusal behaviour is a learned output triggered by specific textual cues, and when those cues are absent or disguised, the behaviour does not activate. The underlying knowledge and the probability of generating the harmful content were never removed; they were only suppressed in contexts the training covered.
Go deeper
- Constitutional AI: Harmlessness from AI Feedback · arXiv · Yuntao Bai et al. · 2022-12-15
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models · arXiv · Samuel Gehman et al. · 2020-09-24
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.