II · THE IDEA · ARTIFICIAL INTELLIGENCE
Constitutional AI for Harmless Assistance
▶ Listen · narrated
Human raters need not label every harmful completion. A fixed list of principles can drive critique, revision, and preference data the model generates for itself.
At a glance
- Core idea
- Critique and revise outputs using a written constitution, not harm labels from humans
- Two stages
- Supervised revision fine-tuning, then RL from AI preference feedback
- Feedback source
- AI judgements against principles (RLAIF), reducing human exposure to toxic text
- Stated aim
- Harmless assistance that still remains helpful
- Hard limit
- Alignment methods face fundamental limitations in large language models
Think of a house style guide pinned to the wall. Instead of asking a person to mark every rude or dangerous draft, you ask the writer to read the guide, list what they got wrong, and rewrite the piece. You then train them on those cleaner rewrites. Later, when two drafts appear, another copy of the same guide is used to pick the better one, and that choice trains a scoring habit that shapes future writing.
Constitutional AI works like that for language models. A constitution is a short list of written principles. The model drafts an answer, critiques it against those principles, and revises it. Those revisions teach the model in a supervised stage. Afterwards, an AI judge compares pairs of answers using the same principles, and reinforcement learning pushes the model toward the preferred ones. People still write the principles and check the system; they do much less direct labelling of harmful text.
Constitutional AI (CAI; Bai et al., arXiv 2212.08073) separates harmlessness supervision into (1) a supervised critique–revision phase and (2) RLAIF. In phase 1, a policy generates a response y0 to prompt x; a critique c is sampled conditioned on constitutional principle(s) P; a revision y' is sampled conditioned on (x, y0, c, P). Fine-tuning on y' yields a policy already biased toward principle-consistent refusals and rewrites. In phase 2, pairwise preferences are labelled by a feedback model instructed with P; a preference/reward model is fit to those labels; RL updates the policy against that reward while a helpfulness signal (often still partly human) is preserved so harmlessness does not collapse into blanket refusal.
Mechanically this replaces much human harm preference data with principle-conditioned chain-of-thought critique and AI preference judgements. Failure modes track the constitution’s coverage and the feedback model’s reliability: omitted harms are under-penalised; conflicting principles yield unstable preferences; judge errors propagate into the reward model. Wolf et al. (arXiv 2304.11082) argue that alignment methods of this class face fundamental limitations in LLMs — behavioural objectives optimised on finite preference data do not entail robust guarantees under adversarial prompting or distribution shift. CAI should therefore be treated as a scalable labelling and shaping pipeline, not as a completeness result for alignment.
Look closer
The constitution is a plain list of principles
Principles are short natural-language rules the model is asked to apply: prefer responses that avoid assisting serious crimes, that respect autonomy, that refuse certain categories of harm, and so on. The same principle text is reused at critique time and at preference-judgement time. Nothing mystical sits behind the word constitution; it is a fixed prompt-facing policy the lab chose and can inspect, edit, or extend. Which principles appear, and how they are worded, directly shapes which failures get penalised.
Critique precedes revision in the supervised stage
The pipeline does not jump straight to a safer answer. The model is first prompted to write a critique of its own draft against one or more constitutional principles, often with chain-of-thought reasoning. Only then is it asked to produce a revised response that addresses the critique. Those revised responses form a supervised dataset used to fine-tune a policy that already tends to answer in the revised style, before any reinforcement learning begins.
Preferences for RL come from AI feedback
In the second stage, pairs of responses are shown to a feedback model that chooses which better satisfies the constitution. Those AI preferences train a preference or reward model, which then supplies the signal for reinforcement learning of the policy. Human labels for harmlessness are largely replaced at this step; humans may still shape helpfulness data or the constitution itself. The acronym used for this pattern is RLAIF — reinforcement learning from AI feedback.
The story
Constitutional AI, as described in the 2022 Anthropic preprint, is a training recipe for making assistants less harmful without sending large volumes of toxic model output to human labelers. The central move is simple to state: write down a constitution — a set of natural-language principles — and train the model to critique and revise its own responses in light of those principles. The revised behaviour then becomes the main source of supervision for harmlessness.
The method has two stages. In the supervised stage, the model produces an initial reply to a prompt that may elicit harmful content. It is next asked to critique that reply against the constitution, and then to rewrite the reply so the critique is addressed. The rewritten replies are collected into a dataset, and the policy is fine-tuned on them. After this stage the model already tends to give answers in the revised register: more refusal where the principles demand it, less assistance on disallowed requests, and explicit reasoning patterns that echo the critique step.
In the reinforcement learning stage, the lab no longer needs humans to rank harmful versus harmless completions for every comparison. Instead, a feedback model judges which of two responses better conforms to the constitutional principles. Those AI-generated preference labels train a preference model, and reinforcement learning updates the policy against that signal. This is the RLAIF pattern: reinforcement learning from AI feedback rather than from human preference labels on the harm axis. The stated goal is a policy that remains helpful on ordinary requests while becoming harder to push into assisting clear harm.
The constitution itself is not learned in secret. It is a written artefact the trainers supply. That makes the target behaviour inspectable in a way a pure black-box reward model is not: if a failure mode is missing from the principles, the critique and preference steps have no instruction to penalise it. Conversely, awkward or conflicting principles can pull the policy in inconsistent directions. The method therefore shifts part of the alignment problem from labelling individual transcripts to drafting and maintaining the principle list.
A separate line of work, including the 2023 preprint on fundamental limitations of alignment in large language models, argues that alignment techniques of this family still face structural bounds. Behavioural fine-tuning and preference optimisation change what the model tends to say under ordinary prompting; they do not yield a guarantee against capable models that can be steered, jailbroken, or used in dual-use settings outside the training distribution. Constitutional AI is best read as an engineering response to the cost and trauma of human harm labelling, and as a way to scale a principle-shaped training signal — not as a proof that the resulting system is aligned in every deployment condition.
What the recipe does show, within the scope of the paper, is a closed loop: principles in natural language, self-critique, revision, AI preference judgements against the same principles, and a policy updated from that loop. The human effort moves upstream, into choosing the constitution and evaluating the overall system, rather than sitting in the path of every toxic sample.
Why it mattered then
When the Constitutional AI work appeared, scaling assistants had made human preference labelling for harmlessness both expensive and personally costly: raters had to read and rank disturbing model outputs at volume. Replacing much of that harm labelling with AI feedback against a written constitution was a direct response to that bottleneck. It also fitted a wider shift toward training recipes that could be iterated inside the lab without a proportional increase in human exposure to toxic text. The method offered a concrete pipeline — critique, revise, then RLAIF — at a moment when labs needed harmlessness training to keep pace with capability increases, and when the only widely deployed alternative was still large-scale human red-teaming and preference collection.
Why it matters now
Principle-shaped self-critique and AI feedback remain part of how many safety stacks are discussed and built: written policies, automated judges, and revision loops are cheaper to rerun than full human labelling campaigns. At the same time, work on fundamental limitations of alignment cautions that such loops do not remove jailbreaks, goal misgeneralisation, or dual-use risk. The practical question is no longer whether CAI-style training can reduce certain refusal and assistance failures on-distribution; it is how much weight to place on constitution text, AI judges, and RL stages when the underlying model is capable enough that failures concentrate in the tail. Reading the original recipe beside the limitations arguments keeps both the engineering gain and the ceiling in view.
The surprising detail
The same natural-language principles do double duty: they prompt the critique-and-revision stage and later serve as the yardstick for AI preference labels. A change in wording therefore propagates through supervised data and through the reward signal without any new human harm annotations. That economy is the point of the design — and also its sensitivity surface. Separately, the limitations preprint stresses that even strong behavioural alignment need not bind a model under adversarial or out-of-distribution pressure, so the elegance of the self-improvement loop should not be mistaken for a completeness proof.
What is disputed
The CAI preprint reports the training recipe and empirical behaviour under its evaluations; it does not establish that constitutional training removes jailbreaks or dual-use risk. The 2023 limitations work argues alignment techniques face fundamental bounds in LLMs. Where those bounds fall for any given deployed system remains disputed and evaluation-dependent.
Remember this
Constitutional AI replaces much human harm labelling with self-critique and AI preferences against a written principle list — powerful scaling, not a guarantee against fundamental alignment limits.
Test yourself
In Constitutional AI, humans still choose the constitution, yet the method claims to reduce human labels for harmlessness. Which two training signals are produced by the model (or an AI feedback model) rather than by human harm raters, and what does each stage do to the policy?
First, revised responses from the critique-and-revision loop form a supervised fine-tuning dataset, so the policy imitates already-rewritten, principle-shaped answers. Second, AI preference judgements between response pairs — scored against the same constitution — train a preference or reward model for reinforcement learning (RLAIF), further steering the policy toward principle-consistent behaviour. Humans may still supply helpfulness data or the principles themselves; what is largely removed is human ranking of toxic completions for the harmlessness objective.
Go deeper
- [2212.08073] Constitutional AI: Harmlessness from AI Feedback · arxiv.org
- [2304.11082] Fundamental Limitations of Alignment in Large Language Models · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.