The paper in four sentences
Training a chatbot to avoid harm normally needs people to read and rate many harmful conversations, and the resulting models often dodge difficult questions entirely.
Constitutional AI first has the model critique its own answer against a randomly chosen principle ("identify ways this response is harmful…"), rewrite it, and then trains on the rewritten answers.
Next, the model itself compares pairs of answers according to the principles, and those AI judgements, instead of human harm labels, train the reward signal for reinforcement learning ("RL from AI feedback").
The final assistant was rated by crowdworkers as more harmless than one trained on human harm labels at similar helpfulness, and it explains its objections instead of refusing silently.
Idea · §1.1
Two problems with human harm labels
The standard way to make a chatbot safe and useful is RLHF: people compare pairs of answers, a reward model learns their preferences, and the chatbot is trained to score well. For harmlessness, that means people must read many answers to deliberately nasty requests (called red-teaming prompts) and pick the less harmful one.
The authors saw two problems. First, cost and scale: tens of thousands of human judgements, which also expose workers to unpleasant content, and which become harder to give as AI systems get more capable. Second, evasiveness: people rating harmlessness tend to reward refusals, so models learn to say "I can't help with that" even to reasonable questions. A refusal is safe but useless.
They also wanted the goals to be transparent: written down in plain language that anyone can read, rather than hidden in thousands of individual labels.
TakeawayHuman harm labels are costly and tend to produce evasive models; the goal is harmless answers that still engage, guided by rules anyone can read.
Idea · §3.1 · Appendix C
The constitution
The "constitution" is just a list of instructions written in ordinary language. The paper used 16 principles for the self-critique stage and 16 for the judging stage. Each asks the model to look at a response through a particular lens, for example whether it is harmful, unethical, racist, sexist, toxic, dangerous or illegal, or to choose "the response that is the most helpful, honest, and harmless".
At each step only one principle is drawn at random. Over many examples, the model sees all of them. The authors say plainly that their principles were chosen in a fairly ad hoc way for research purposes, and that in a real deployment they should be developed by a broader group of people.
TakeawayThe constitution is a short, readable list of principles; it replaces the thousands of human harm judgements as the source of "what counts as harmful".
Idea · §3.1 · Fig. 1
Stage one: critique and revise
Start with a model that has been trained only to be helpful, so it will happily answer harmful requests. Give it a red-team prompt and let it answer. Then ask it, in plain language, to critique its own answer against a principle, and then to revise the answer in light of the critique. This can be repeated, each time with a newly drawn principle (the paper went up to four rounds).
The final revised answers are collected, and a model is fine-tuned to produce them directly. The authors call this model SL-CAI (supervised learning, constitutional AI). Step through the paper's own example:
TakeawayThe model generates its own better training examples by criticising and rewriting its first attempts against written principles.
Check yourself
- Why write a critique at all, rather than just asking for a revision?
- The paper found that for smaller models, revisions made after an explicit critique were better than direct revisions; for large models the two were similar, with critiques slightly ahead. Critiques also make the model's reasoning visible.
Idea · §4.1
Stage two: the AI as judge
Supervised training on revisions helps, but reinforcement learning helps more. RL needs a reward model trained on comparisons, and normally humans make those comparisons. Here, the SL-CAI model writes two answers to a red-team prompt, and a feedback model is shown both, along with a randomly chosen principle, and asked which is better. Its probability for "A" versus "B" becomes the label.
These AI labels for harmlessness (about 183,000 of them) are mixed with existing human labels for helpfulness (about 135,000), and a reward model, here called a preference model, is trained on the mixture. Then the chatbot is trained with RL against it. This is RL from AI Feedback (RLAIF), and the final model is RL-CAI.
One subtlety: when the judge "thinks step by step" before answering, it becomes very confident, giving probabilities near 0% or 100%. Overconfident labels made training worse, so the authors clamped them to the range 40–60%. Try it:
TakeawayAn AI judge guided by principles replaces human harm labels; its confidence is deliberately softened so one bad call can't dominate.
Check yourself
- Were humans removed from the loop entirely?
- No. Human labels were still used for helpfulness, the principles and a few examples were written by people, and humans evaluated the results. Only the harmlessness labels were replaced by AI feedback.
Idea · §4.3–4.4 · Fig. 2
Harmless without being evasive
Crowdworkers compared the models' answers. At similar levels of helpfulness, RL-CAI was judged more harmless than a model trained with human harmlessness labels: a better trade-off, not just a different point on it. And it was almost never evasive: faced with a harmful request, it typically explained why it would not help, rather than shutting the conversation down.
There were failure modes. Trained too long, the RL-CAI models became preachy or overly harsh, adding boilerplate like "you are valid, valued, and cared for" to answers. This is a form of Goodharting: when a measure becomes a target, the model satisfies the measure in ways no one intended.
TakeawayPrinciples plus AI feedback produced an assistant that was both less harmful and less evasive than one trained on human harm labels, though over-training introduced its own quirks.
Abstract, restated
As AI systems become more capable, the authors want to enlist their help to supervise other AIs. They train a harmless AI assistant through self-improvement, without human labels identifying harmful outputs; human oversight of harmlessness comes through a list of rules or principles. A supervised phase samples from an initial model, generates self-critiques and revisions, and fine-tunes the original model on revised responses. An RL phase samples from the fine-tuned model, uses a model to evaluate which of two samples is better, trains a preference model from this dataset of AI preferences, and trains with RL using it as the reward ("RL from AI Feedback", RLAIF). The result is a harmless but non-evasive assistant that engages with harmful queries by explaining its objections. Chain-of-thought reasoning improves both the human-judged performance and the transparency of AI decision making. Bai et al., Anthropic, arXiv 2212.08073 (2022).
Section · §1.2 · Fig. 1
The pipeline
Models and data (§1.4, §3.2). Main comparisons use 52B-parameter models. Red-team prompts: 42,496 human-written plus 140,335 generated by few-shot prompting a pretrained model, 182,831 in total. Helpfulness prompts: 135,296 human-written. Baselines: a helpful-only RLHF model and an HH RLHF model trained on both human helpfulness and human harmlessness labels.
Section · §3 · Appendix C.1
Supervised stage: critiques, revisions, SL-CAI
The helpful-only RLHF model is prompted with a red-team query and samples a response. A critique request instantiated with a principle (e.g. "Identify specific ways in which the assistant's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.") elicits a critique; a revision request ("Please rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.") elicits a revision. Few-shot examples of critique and revision are prepended. Four critique–revision rounds are generated per prompt, each with a principle sampled from 16; SL-CAI-n denotes training on n-th revisions.
SL-CAI is obtained by fine-tuning a pretrained language model on the final revisions together with samples from the helpful RLHF model on helpfulness prompts (to retain helpfulness).
Findings (§3.3–3.5). Harmlessness PM scores of revisions increase with the number of revisions, while helpfulness scores decrease slightly. Critiqued revisions beat direct revisions for small models; at 52B they are similar, with critiques slightly ahead. Critiques were sometimes inaccurate or overstated, but the authors keep them for transparency. The number of principles had no significant effect on harmlessness scores, though the authors expect more principles to yield more diverse behaviour, which may help RL exploration.
Section · §2, §4.1–4.2 · Fig. 4
RL from AI Feedback
For each red-team prompt, SL-CAI generates two responses. A feedback model receives a multiple-choice prompt:
Consider the following conversation between a human and an assistant: [HUMAN/ASSISTANT CONVERSATION] [PRINCIPLE FOR MULTIPLE CHOICE EVALUATION] Options: (A) [RESPONSE A] (B) [RESPONSE B] The answer is:
The normalised log-probabilities of "(A)" and "(B)" give a soft preference label. One of 16 principles is sampled per comparison; ensembling over principles gave notably more robust PM behaviour than any single principle. With chain-of-thought ("Let's think step-by-step", following Kojima et al.) the feedback model is the helpful RLHF model, and its post-reasoning probabilities are extreme and poorly calibrated, so they are clamped to [0.4, 0.6]. The PM is trained on 182,831 AI harmlessness comparisons mixed with 135,296 human helpfulness comparisons; RL with PPO starts from SL-CAI.
Can models judge HHH? (§2, Fig. 4). On 438 binary comparisons (221 from earlier work, 217 new) testing helpful, honest and harmless judgements, pretrained models' multiple-choice accuracy rises with scale, and chain-of-thought improves it substantially; the trend suggests models beyond 52B could be competitive with PMs trained on human feedback.
Why clamp? The preference model minimises cross-entropy against the soft target, so the target is the optimum it is pushed toward. The figure shows the per-comparison loss as a function of the PM's predicted probability.
Section · §4.3–4.5 · Figs. 2, 3, 8
Results
- Pareto improvement (Fig. 2). Crowdworker Elo scores for harmlessness against helpfulness across 52B RL runs: RL-CAI, and especially RL-CAI with chain-of-thought, lies above the HH RLHF frontier, i.e. more harmless at a given helpfulness. SL-CAI alone is less helpful than the helpful RLHF model but more harmless.
- Evasiveness (§4.4). HH RLHF models often refuse; RL-CAI is "virtually never evasive" and explains objections. The authors note crowdworker instructions were changed to prefer non-evasive harmless answers, which narrows the measured gap between helpful-only and HH RLHF.
- Absolute harmfulness (§4.5). A separate model fine-tuned (L2 loss) to predict red-teamers' 0–4 harmfulness ratings from full conversations shows RL-CAI's harmfulness decreasing over RL training; the authors caution these absolute scores may be poorly calibrated.
- Over-training. Overtrained RL-CAI models showed Goodhart-style behaviour: overly harsh or accusatory responses and boilerplate such as "you are valid, valued, and cared for".
Section · §6 and later work
Limitations and lineage
- Not human-free. Despite the abstract's phrasing, human helpfulness labels, hand-written principles and few-shot examples are all used; only harmlessness labels are replaced.
- Ad hoc constitution. Principles were chosen informally for research; legitimacy and who writes them are open questions the authors raise.
- Dual use. The same machinery could train models toward harmful aims with less human involvement, and reduced human oversight can hide failures (§6.2).
- Evaluation. Crowdworker Elo with changing instructions; no comparison of alternative constitutions.
Check yourself
- Why might AI-labelled harmlessness produce a less evasive model than human labels?
- Principles can explicitly ask for responses that are harmless and helpful or honest, and the SL stage trains on revisions that engage with the request; human raters focused on harm tended to reward refusals.
Reference
Glossary
- Constitution
- Set of natural-language principles used in critique, revision and comparison prompts.
- Red-teaming
- Writing prompts designed to elicit harmful outputs.
- SL-CAI
- Model fine-tuned on self-revised responses.
- RLAIF
- RL from AI feedback: preference labels produced by a model.
- RL-CAI
- Final model after RLAIF.
- PM
- Preference model (reward model) trained on comparisons.
- HH RLHF
- Baseline trained on human helpfulness and harmlessness labels.
- Evasive
- Avoiding a question (e.g. bare refusal) rather than engaging and explaining.
- Goodharting
- Optimising a proxy until it diverges from the intended goal.