Study guide · Safety & security · 2022

Constitutional AI: Harmlessness from AI Feedback

Yuntao Bai et al. · arXiv

Can an AI assistant be trained to avoid harmful behaviour without humans labelling thousands of harmful conversations? Bai and colleagues at Anthropic replace those labels with a short written list of principles, a "constitution", which the model uses to criticise and rewrite its own answers and then to judge pairs of answers. The result was less harmful than a model trained on human harm labels and, importantly, less evasive.

The paper in four sentences

Training a chatbot to avoid harm normally needs people to read and rate many harmful conversations, and the resulting models often dodge difficult questions entirely.

Constitutional AI first has the model critique its own answer against a randomly chosen principle ("identify ways this response is harmful…"), rewrite it, and then trains on the rewritten answers.

Next, the model itself compares pairs of answers according to the principles, and those AI judgements, instead of human harm labels, train the reward signal for reinforcement learning ("RL from AI feedback").

The final assistant was rated by crowdworkers as more harmless than one trained on human harm labels at similar helpfulness, and it explains its objections instead of refusing silently.

Idea · §1.1

Two problems with human harm labels

The standard way to make a chatbot safe and useful is RLHF: people compare pairs of answers, a reward model learns their preferences, and the chatbot is trained to score well. For harmlessness, that means people must read many answers to deliberately nasty requests (called red-teaming prompts) and pick the less harmful one.

The authors saw two problems. First, cost and scale: tens of thousands of human judgements, which also expose workers to unpleasant content, and which become harder to give as AI systems get more capable. Second, evasiveness: people rating harmlessness tend to reward refusals, so models learn to say "I can't help with that" even to reasonable questions. A refusal is safe but useless.

They also wanted the goals to be transparent: written down in plain language that anyone can read, rather than hidden in thousands of individual labels.

Everyday pictureTraining a new employee by having a supervisor judge every one of their emails is slow, and the employee may learn to play safe by replying "please contact another department" to everything. Giving them a written code of conduct and asking them to check their own drafts against it scales better and is easier to inspect.

TakeawayHuman harm labels are costly and tend to produce evasive models; the goal is harmless answers that still engage, guided by rules anyone can read.

Idea · §3.1 · Appendix C

The constitution

The "constitution" is just a list of instructions written in ordinary language. The paper used 16 principles for the self-critique stage and 16 for the judging stage. Each asks the model to look at a response through a particular lens, for example whether it is harmful, unethical, racist, sexist, toxic, dangerous or illegal, or to choose "the response that is the most helpful, honest, and harmless".

At each step only one principle is drawn at random. Over many examples, the model sees all of them. The authors say plainly that their principles were chosen in a fairly ad hoc way for research purposes, and that in a real deployment they should be developed by a broader group of people.

TakeawayThe constitution is a short, readable list of principles; it replaces the thousands of human harm judgements as the source of "what counts as harmful".

Idea · §3.1 · Fig. 1

Stage one: critique and revise

Start with a model that has been trained only to be helpful, so it will happily answer harmful requests. Give it a red-team prompt and let it answer. Then ask it, in plain language, to critique its own answer against a principle, and then to revise the answer in light of the critique. This can be repeated, each time with a newly drawn principle (the paper went up to four rounds).

The final revised answers are collected, and a model is fine-tuned to produce them directly. The authors call this model SL-CAI (supervised learning, constitutional AI). Step through the paper's own example:

All text is quoted from the worked example in §3.1 of the paper. The critique and revision requests are fixed templates filled with a principle; the critique and revision are written by the model itself. Only the final revision becomes training data.
Everyday pictureA student writes an essay, then rereads it with one item from the marking guide in mind ("check your argument is fair to the other side"), writes notes on what is wrong, and redrafts. The redrafts, not the first drafts, go into the portfolio.

TakeawayThe model generates its own better training examples by criticising and rewriting its first attempts against written principles.

Check yourself
Why write a critique at all, rather than just asking for a revision?
The paper found that for smaller models, revisions made after an explicit critique were better than direct revisions; for large models the two were similar, with critiques slightly ahead. Critiques also make the model's reasoning visible.

Idea · §4.1

Stage two: the AI as judge

Supervised training on revisions helps, but reinforcement learning helps more. RL needs a reward model trained on comparisons, and normally humans make those comparisons. Here, the SL-CAI model writes two answers to a red-team prompt, and a feedback model is shown both, along with a randomly chosen principle, and asked which is better. Its probability for "A" versus "B" becomes the label.

These AI labels for harmlessness (about 183,000 of them) are mixed with existing human labels for helpfulness (about 135,000), and a reward model, here called a preference model, is trained on the mixture. Then the chatbot is trained with RL against it. This is RL from AI Feedback (RLAIF), and the final model is RL-CAI.

One subtlety: when the judge "thinks step by step" before answering, it becomes very confident, giving probabilities near 0% or 100%. Overconfident labels made training worse, so the authors clamped them to the range 40–60%. Try it:

Top bar: what the AI judge says. Bottom bar: the label the preference model is trained to reproduce. With clamping at 40–60%, even a 97%-sure judge only teaches "A is somewhat better", so a confident but wrong judgement does limited damage. The paper reports 40–60% worked better than 20–80% (§4.1).

TakeawayAn AI judge guided by principles replaces human harm labels; its confidence is deliberately softened so one bad call can't dominate.

Check yourself
Were humans removed from the loop entirely?
No. Human labels were still used for helpfulness, the principles and a few examples were written by people, and humans evaluated the results. Only the harmlessness labels were replaced by AI feedback.

Idea · §4.3–4.4 · Fig. 2

Harmless without being evasive

Crowdworkers compared the models' answers. At similar levels of helpfulness, RL-CAI was judged more harmless than a model trained with human harmlessness labels: a better trade-off, not just a different point on it. And it was almost never evasive: faced with a harmful request, it typically explained why it would not help, rather than shutting the conversation down.

There were failure modes. Trained too long, the RL-CAI models became preachy or overly harsh, adding boilerplate like "you are valid, valued, and cared for" to answers. This is a form of Goodharting: when a measure becomes a target, the model satisfies the measure in ways no one intended.

Later refinementAI feedback became a standard ingredient of model training. Anthropic later experimented with a constitution drafted with public input (Collective Constitutional AI, 2023) and published the principles used for Claude. Other labs adopted RLAIF for some preference labels, and Lee et al. (2023) found RLAIF comparable to RLHF on summarisation and dialogue. DPO showed the preference labels can be used without an RL loop at all.

TakeawayPrinciples plus AI feedback produced an assistant that was both less harmful and less evasive than one trained on human harm labels, though over-training introduced its own quirks.

Follow the idea

Built on: InstructGPT (RLHF) (2022)