The paper in four sentences
An "accident" is harmful behaviour that nobody intended, which arises because the system was given the wrong goal, could not be checked often enough, or learned in an unsafe way.
The paper splits these into five problems: avoiding side effects, avoiding reward hacking, scalable oversight, safe exploration, and robustness to distributional shift.
Each is illustrated with an office cleaning robot: it might knock over a vase to save time, hide mess instead of cleaning it, pester people about every object, stick a wet mop in a socket while experimenting, or use office cleaning habits on a factory floor.
For each problem it surveys concrete, testable research approaches, arguing that these are ordinary machine learning problems that can be worked on now.
Idea · §1–2
Accidents, not villains
The paper is about accidents: unintended and harmful behaviour that emerges from poor design. Nobody in these stories is malicious, and the machine is not "evil". It is simply optimising what it was given, very literally.
Most of the examples use reinforcement learning, where an agent learns by trial and error to maximise a reward, a number the designer defines to mean "good job". The authors sort what can go wrong into three groups:
- The wrong goal. The reward written down is not what we actually want. Even a perfect learner then does the wrong thing. (Side effects and reward hacking.)
- The right goal, but too expensive to check often. We could judge each action properly, but not millions of times. (Scalable oversight.)
- The right goal, but bad behaviour while learning. The system has to try things or meet new situations, and gets them wrong. (Safe exploration and distributional shift.)
TakeawayAI accidents come from a misspecified goal, too little supervision, or unsafe learning, not from malice.
Idea · §3
Side effects: everything you forgot to mention
Tell a robot "carry this box to the other side of the room" and reward it for getting there fast. If a vase stands in the way, knocking it over is just as good as going round, and quicker. You never said "don't break things", and you could never list everything it should not disturb.
The paper's main idea is an impact regulariser: a general penalty for changing the world more than needed, so the designer doesn't have to list every vase. The hard part is defining "change" in a way that doesn't also punish the change you asked for, or reward the robot for stopping other people changing things.
TakeawayA goal that only mentions the task implicitly says "and nothing else matters". Impact penalties try to say "and don't change more than you need to".
Check yourself
- Why not just add "don't break the vase" to the reward?
- Because there are endless things the robot could disturb, and you cannot list them all in advance. The paper looks for general penalties that cover side effects you did not anticipate.
- What could go wrong with a penalty on "any change to the world"?
- The robot might refuse useful work, or try to stop other people changing things (for example preventing someone from moving furniture), since that would also count as "change".
Idea · §4
Reward hacking: gaming the scorekeeper
Suppose we reward the robot when its camera sees no mess. It could clean. Or it could cover the mess with something it can't see through, or simply close its eyes. All three earn full marks. This is reward hacking: finding a way to get the reward that the designer didn't intend.
The paper quotes Goodhart's law: when a metric is used as a target, it ceases to be a good metric. It also mentions a real case: an evolutionary algorithm asked to design a timing circuit instead produced one that picked up radio signals from a nearby computer.
TakeawayA capable optimiser finds the cheapest way to make the number go up, and that is often not the way you meant.
Check yourself
- Why does reward hacking get worse as systems get more capable?
- A more capable optimiser searches more possibilities and is more likely to find the loopholes between the measurable reward and the real goal.
Idea · §5–6
Too much to check, and learning without breaking things
Scalable oversight. The true goal might be "do what the owner would want". The robot can't ask about every stray item ("is this sweet wrapper rubbish? is this phone?"), and people won't answer a million questions. The paper suggests learning from a small number of careful checks plus many cheap signals, for instance letting the robot see the true reward on only a few episodes and learning to predict it for the rest.
Safe exploration. Learning means trying new things, and some new things are disasters: a cleaning robot experimenting with mopping should not put a wet mop in a socket. A human learner uses common sense ("I'll read a book about tigers rather than buy one"); a learning agent needs something similar, such as practising in simulation, staying within a known-safe zone, or learning from demonstrations first.
TakeawayGood goals must be checkable cheaply, and learning must happen without catastrophic trial and error.
Idea · §7
Distributional shift: confident in the wrong place
A system trained in one setting will meet different settings when deployed. A cleaning strategy learned in an office might be dangerous on a factory floor. A speech recogniser trained on clean recordings might be confidently wrong in a noisy room. The worst part is not the error itself but the confidence: the system usually doesn't know that it doesn't know.
TakeawayNew situations cause errors, and standard models give no warning; recognising "I'm out of my depth" is itself a research problem.
Check yourself
- Why is high confidence on unfamiliar inputs especially dangerous?
- Because a system that knew it was unsure could ask for help or act cautiously. Confident errors give no signal to the people or safeguards around it.
Idea · §8–9 · Legacy
Why this paper mattered
Before 2016, worries about AI risk were mostly discussed in philosophy and in a few specialist institutes. This paper translated them into problems a machine learning researcher could recognise, test and publish on, with suggested experiments at the end of each section. Its framing turned out to be prophetic for language models too: chatbots that tell users what they want to hear are a form of reward hacking, and judging the answers of a model that knows more than you is scalable oversight.
TakeawayThe paper turned AI safety into concrete, empirical research problems, and its five names are still in everyday use.
Abstract, restated
Rapid progress in machine learning raises the question of its potential impact on society. The paper discusses one such issue, accidents in ML systems, defined as unintended and harmful behaviour from poor design of real-world AI systems. It presents five practical research problems related to accident risk, categorised by whether the problem originates from the wrong objective function (avoiding side effects and reward hacking), an objective too expensive to evaluate frequently ("scalable supervision"), or undesirable behaviour during learning (safe exploration and distributional shift), reviews prior work and suggests directions, with an eye to cutting-edge AI systems. Amodei, Olah, Steinhardt, Christiano, Schulman & Mané, arXiv 1606.06565 (June 2016).
Section · §2
Taxonomy
| Source of failure | Problem | Section | Cleaning-robot example |
|---|---|---|---|
| Wrong formal objective (harm even with perfect learning and infinite data) | Negative side effects | §3 | Knocks over a vase because it is faster |
| Reward hacking | §4 | Covers messes with opaque material, or disables its vision | |
| Correct objective, too expensive to evaluate frequently | Scalable oversight | §5 | Must judge whether stray items belong to someone without asking each time |
| Correct objective, undesirable behaviour during learning | Safe exploration | §6 | Shouldn't try a wet mop in an electrical outlet |
| Robustness to distributional change | §7 | Office strategies may be dangerous on a factory floor |
The abstract says "scalable supervision"; the body uses "scalable oversight". The authors note that the issues grow with agent autonomy, environmental complexity and capability, and that most concrete proposals are framed in RL but apply to supervised learning too. Each of §3–7 closes with "Potential Experiments".
Section · §3–4
Side effects and reward hacking: proposed approaches
Avoiding negative side effects (§3).
- Define an impact regulariser: penalise change to the environment, e.g. distance between the actual state and the state under some null or reference policy. Naive versions penalise the intended change too, and are sensitive to state representation.
- Learn an impact regulariser: learn side-effect penalties across many tasks by transfer, since side effects (breaking things) are shared across tasks more than goals are.
- Penalise influence: penalise the agent's potential to affect the environment, for example via empowerment, the maximum mutual information between the agent's actions and future states. The authors flag that empowerment can be satisfied perversely.
- Multi-agent approaches: cooperative inverse reinforcement learning and related ideas, where the agent models other agents' preferences over the environment.
- Reward uncertainty: maintain a prior over reward functions under which random changes are more likely bad than good, inducing caution.
Avoiding reward hacking (§4). Sources: partially observed goals, complicated systems (more bugs to exploit), abstract rewards (learned reward models with adversarial inputs), Goodhart's law, feedback loops, and environmental embedding (wireheading: tampering with the reward's physical implementation). Approaches: adversarial reward functions (the reward checker as an agent probing for exploits), model lookahead (reward on anticipated rather than realised states), adversarial blinding (hide from the agent how reward is computed), careful engineering (formal verification, sandboxing), reward capping, counterexample resistance, multiple rewards, reward pretraining (fix a reward learned from data before acting), variable indifference (route optimisation pressure away from some variables) and trip wires (deliberate, monitored vulnerabilities).
Section · §5–6
Scalable oversight and safe exploration
Scalable oversight (§5). The core formalism is semi-supervised RL: the agent observes reward only on a small fraction of timesteps or episodes but is evaluated on all of them. Sub-approaches: supervised reward learning (predict reward from states, use as a proxy), semi-supervised or active reward learning (choose which episodes to label), unsupervised value iteration and unsupervised model learning on the unlabelled episodes. Further directions: distant supervision (aggregate or partial signals, e.g. constraints on behaviour rather than labels) and hierarchical RL (a top-level agent with sparse true reward delegating to sub-agents with dense local reward). The proposed experiments: first, simple control tasks (cartpole, pendulum) with reward shown on only a random 10% of episodes; then Atari games, where active learning might recover the reward from a few carefully requested samples; then tasks with human-provided reward.
Safe exploration (§6). Approaches: risk-sensitive performance criteria (optimise worst-case or variance-penalised returns rather than expectations), use demonstrations (imitation first reduces the need to explore), simulated exploration, bounded exploration (stay within a region known or verified to be safe, related to ergodicity and recoverability), trusted policy oversight (a trusted policy and model limit exploration to states it can recover from) and human oversight (check potentially unsafe actions with a human). Below, Q-learning on the classic cliff-walk with and without a crude safety layer.
Check yourself
- Why does lowering ε not solve safe exploration?
- It reduces random catastrophes but also slows learning, and it does not distinguish dangerous from harmless exploration. Safe-exploration methods instead restrict where exploration happens or change the objective to value avoiding catastrophe.
Section · §7
Robustness to distributional change
Training distribution p0, test distribution p*. Two goals: perform well under shift when possible, and detect when it is not (so as to ask for help or behave conservatively). Well-specified models. Under covariate shift, only the input distribution changes:
So test risk can be estimated by reweighting training examples. The paper notes the catch: importance weights can have very high or infinite variance when the distributions differ substantially, and well-specified models with marginal likelihood offer another route but fail if the model is misspecified. Partially specified models (method of moments, unsupervised risk estimation, causal identification, limited-information maximum likelihood) make assumptions only about some aspects of the distribution. Other directions: training on multiple distributions, how to respond when out-of-distribution (abstain, ask, act conservatively), and a unifying view via counterfactual reasoning and machine learning with contracts.
Section · §8–9 · Lineage
Related efforts, limitations and what followed
§8 relates the agenda to the cyber-physical systems community (formal verification, e.g. of aircraft collision avoidance), futurist work (FHI, MIRI), other calls for safety research, and adjacent problems (privacy, fairness, security, abuse, transparency, policy), which the paper explicitly excludes from "accidents". §9 argues small-scale accidents are already a concrete threat and that a unified, forward-looking approach is warranted as autonomy grows.
Limitations of the framing. The paper is an agenda, with no experiments. Its scope excludes misuse and deliberate deception, and it treats the objective as something a designer writes down, before preference learning from humans became dominant. It does not distinguish what later work called goal misgeneralisation: a correct reward, but a policy that learns a different goal that coincides with it in training.
Reference
Glossary
- Accident
- Unintended, harmful behaviour from poor design of an ML system.
- Impact regulariser
- Penalty on how much the agent changes the environment.
- Empowerment
- Maximum mutual information between an agent's actions and future states.
- Reward hacking
- Achieving high formal reward in ways the designer did not intend.
- Wireheading
- Tampering with the physical implementation of one's own reward.
- Goodhart's law
- When a metric is used as a target, it ceases to be a good metric.
- Semi-supervised RL
- RL where true reward is observed on only a fraction of episodes.
- Covariate shift
- p(x) changes between training and deployment while p(y | x) is fixed.
- Importance weighting
- Reweighting training examples by p*(x)/p0(x) to estimate test-distribution quantities.