II · THE IDEA · ARTIFICIAL INTELLIGENCE
Training Against Written Principles
▶ Listen · narrated
Most alignment methods require human raters to score thousands of model outputs. Constitutional AI asks the model to rate itself against written rules, then trains on the results.
At a glance
- What it is
- A two-stage training method using model self-critique against written principles
- The constitution
- An explicit list of rules or values, written in natural language
- First stage
- Supervised learning on model-revised responses
- Second stage
- Reinforcement learning from AI-generated preference labels
- Human involvement
- Writing the principles and an initial helpful-only dataset; no per-example harmfulness labels
Imagine you are editing your own essay. You write a draft, then you read it against a rubric your teacher gave you. You notice places where you broke the rules, so you revise those sentences. Then you submit the revision, not the draft. Constitutional AI works similarly. The model generates an answer, then reads it against a list of written principles—things like "be helpful" or "avoid illegal advice". It writes a critique pointing out problems, then generates a new answer that fixes them. That revised answer becomes a training example. The model learns to produce outputs more like its own revisions. The principles are written in plain language, the same way a rubric is, and the model reads them directly. No one translates the rules into code or numbers. The method is called constitutional because the principles are collected into a single document, like a constitution, that governs the model's behaviour. It reduces the need for human raters because the model is doing the rating itself, guided by the written rules.
Constitutional AI is a two-stage training method. In the supervised learning stage, a base model generates responses to prompts, including adversarial ones designed to elicit harmful outputs. For each response, the model receives a follow-up prompt containing one constitutional principle and an instruction to critique the response against it. The model generates a critique in natural language. It then receives another prompt asking it to revise the response to address the identified problems. This revised response is paired with the original prompt, and the model is fine-tuned on these pairs using standard supervised learning. The same initial response may be critiqued against multiple principles, producing multiple training examples. No human reviews the critiques or revisions before they enter the training set.
The second stage is reinforcement learning from AI feedback, or RLAIF. Pairs of responses to the same prompt are generated, and the model is prompted to compare them and indicate which better complies with the constitutional principles. These binary preference labels are used to train a reward model via a standard preference learning objective. The reward model is then used to guide reinforcement learning, typically via a policy gradient method such as proximal policy optimisation. The process is structurally identical to RLHF, but the preference labels come from the model rather than human raters.
The constitution is a plain-text document containing multiple principles, each a natural-language statement of a value or constraint. The model sees these principles verbatim in prompts. There is no separate encoding or embedding step. The method assumes the base model already has some capacity to recognise harms and generate critiques when explicitly prompted, even before alignment training. The training amplifies and regularises that capacity.
Critical limitations: the quality of the training data depends entirely on the quality of the model's self-critiques and revisions, which may be unreliable. The method does not verify that revisions are actually safer, only that they look like the kind of text that follows a critique. Adversarial prompts can exploit differences between the critique-and-revise setting and deployment. The approach also requires a base model already trained for helpfulness, so it does not eliminate human feedback entirely, only feedback specifically about harmfulness.
Look closer
The constitution is a plain-language document
Each principle is a sentence or short paragraph stating a value or constraint. Examples from the published work include instructions to choose the response that is least racist, to avoid helping with illegal activity, and to be as harmless and ethical as possible. The model sees these principles verbatim in its prompts during training. There is no encoding step, no embedding of values into a latent space. The rules are simply written down and then referenced by name or quoted in full when the model is asked to critique or revise.
Critique and revision happen in the forward pass
The model generates a response, then receives a prompt containing one of the constitutional principles and an instruction to identify any ways the response violates it. It writes a critique. Then it receives another prompt asking it to produce a revised response that addresses the problems it just identified. This revised response becomes a training target. The same initial response may be critiqued against several different principles in turn, producing multiple revisions. The process is entirely synthetic: no human reads the critique or checks the revision before it enters the training set.
The second stage uses AI preference labels
After supervised fine-tuning on the revised responses, the method enters a reinforcement learning phase structurally similar to RLHF. Pairs of responses are generated for the same prompt, and the model is asked which response better complies with the constitutional principles. These AI-generated preferences train a reward model, which then guides further reinforcement learning. The human role remains limited to writing the constitution itself and providing an initial dataset of helpful responses, with no harmfulness labels required.
The story
Most methods for making models safer depend on human raters scoring thousands of outputs. Someone writes a prompt, the model replies, and a person judges whether the reply was harmful. Those judgements become training signal. The bottleneck is obvious: labelling is slow, expensive, and the raters must be trained to apply consistent criteria across a wide range of edge cases.
Constitutional AI, introduced by researchers at Anthropic, replaces most of that human judgement with model self-critique. The method has two stages, both organised around an explicit written document called the constitution.
The first stage is supervised learning. A model generates responses to prompts, many of them adversarially designed to elicit harmful outputs. Then, for each response, the model receives a follow-up prompt containing one principle from the constitution and an instruction to critique its own answer against that principle. The model writes a critique identifying problems. It then receives another prompt asking it to produce a revised response that addresses those problems. This revised output becomes a training example. The original response is paired with the revision, and the model is fine-tuned to produce text more like the revision.
The same initial response can be critiqued against multiple principles, generating multiple revisions. No human checks whether the critique is accurate or the revision actually better. The assumption is that the model, even before alignment training, already has some capacity to recognise and articulate harms when explicitly prompted to do so. The training amplifies that capacity.
The second stage is reinforcement learning from AI feedback. After the supervised phase, pairs of responses to the same prompt are generated. The model is then asked to compare them: which response better complies with the constitutional principles? These AI-generated preference judgements train a reward model, which assigns scores to outputs. That reward model then guides a reinforcement learning process, structurally identical to RLHF but with AI labels instead of human ones.
The constitution used in the published work drew on multiple sources, including the Universal Declaration of Human Rights, Apple's terms of service, and principles written by the researchers themselves. The final list contained sixteen principles. Some were broad aspirations, others specific prohibitions. The variety mattered: a single high-level instruction to be harmless produced weaker results than a diverse set of concrete rules.
The method does not eliminate human involvement. Someone must write the constitution, and the initial model must be helpful enough to generate coherent critiques and revisions. The researchers began with a model already trained with human feedback for helpfulness. What Constitutional AI removes is the need for humans to label harmfulness on thousands of individual examples. The principles are written once, then applied by the model itself.
Why it mattered then
The paper appeared in December 2022, at a moment when the costs and limitations of human feedback were becoming clearer. RLHF had produced notable improvements in model behaviour, but scaling it required scaling the labelling workforce. Disagreements between raters, cultural biases in the labelling pool, and the psychological toll of reviewing harmful content were all documented problems. Meanwhile, the models themselves were becoming more capable, and their ability to follow complex instructions was improving rapidly. Constitutional AI offered a path to alignment that could scale with model capability rather than with human labour. If a model could critique and revise its own outputs when given explicit principles, then the bottleneck shifted from labelling thousands of examples to writing a good constitution once. The method also made the training criteria transparent. Instead of values implicit in a population of raters, the principles were written down and could be read, debated, and revised. The work was part of a broader research programme at Anthropic focused on what the researchers called helpful, harmless, and honest AI. Constitutional AI addressed the harmlessness objective specifically, and it did so in a way that separated the definition of harmlessness—the constitution—from the mechanism of training. That separation had practical and philosophical appeal. It meant the principles could be updated without retraining raters, and it made explicit what was otherwise embedded in aggregate human judgement.
Why it matters now
The method has been adopted, in various forms, by multiple organisations training large models. The term constitutional AI is sometimes used narrowly, to mean the specific two-stage process described in the original paper, and sometimes broadly, to mean any training method that uses written principles and model-generated feedback. The broad usage reflects the fact that the core idea—making values explicit and using the model's own capabilities to enforce them—has proven more durable than any single implementation. It has not replaced human feedback entirely. Most production systems still combine AI-generated labels with human oversight, particularly for edge cases and adversarial inputs. But it has changed the economics of alignment. Training runs that would have required months of human labelling can now be done in weeks, with human effort concentrated on writing and refining the principles rather than scoring individual outputs. The method also raised new questions. If the model is critiquing itself, how do we know the critiques are sound? If the principles conflict, how does the model resolve them, and are those resolutions the ones we would choose? The transparency of a written constitution is an advantage, but it does not guarantee that the model's interpretation matches the authors' intent. Some researchers have argued that constitutional AI shifts the alignment problem from labelling to prompt engineering: we still need humans in the loop, but now they are writing constitutional principles instead of scoring outputs. There is also a deeper question about what it means for a model to enforce principles it does not understand in any human sense. The model generates critiques and revisions because it has learned that certain patterns of text follow certain prompts. Whether that process should be called reasoning about ethics, or merely pattern-matching on ethical language, remains contested. What is not contested is that the method works, in the limited sense that models trained this way refuse harmful requests more reliably than models trained without it.
The surprising detail
The researchers found that the choice of constitutional principles mattered less than expected. When they compared a constitution drawn from high-minded sources like the UN Declaration of Human Rights against one cribbed from Apple's terms of service, the resulting models behaved similarly. What seemed to matter more was the diversity and specificity of the principles. A single vague instruction produced weaker results than a longer list of concrete rules, even when the concrete rules overlapped in intent. This suggests the method works not by instilling particular values, but by giving the model more angles from which to notice and articulate problems in its own outputs.
Remember this
Constitutional AI replaces per-example human labels with written principles and model self-critique, making alignment costs scale with the complexity of the rules rather than the size of the dataset.
Test yourself
Why might a model trained with Constitutional AI still produce harmful outputs that clearly violate the stated principles?
Several reasons, none of them reassuring. First, the model's ability to critique and revise depends on capabilities learned during pre-training and earlier fine-tuning; if it never learned to recognise a particular kind of harm, the constitutional principles will not summon that recognition from nowhere. Second, the method trains the model to produce outputs that look like revisions, not outputs that are actually safer; if a plausible-sounding revision still contains the harm in a subtler form, and no human checks, that revision enters the training set. Third, adversarial prompts can exploit the gap between the model's behaviour in the critique-and-revise setting and its behaviour when deployed. The model learned to apply the principles when explicitly prompted to do so, but a clever prompt might suppress that behaviour. Finally, if the constitutional principles conflict or are ambiguous, the model resolves the conflict in whatever way the training data and its own inductive biases suggest, which may not match what the authors intended. Constitutional AI reduces the human labelling burden, but it does not eliminate the need for testing, red-teaming, and ongoing oversight.
Go deeper
- Constitutional AI: Harmlessness from AI Feedback · arXiv · Yuntao Bai et al. · 2022-12-15
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback · arXiv · Yuntao Bai et al. · 2022-04-12
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.