Study guide · Robotics & reinforcement learning · 2017

Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov · arXiv

Can you get the stability of a trust-region method like TRPO with nothing more than ordinary gradient steps? PPO says yes: clip the objective so that an update gains nothing from pushing any action's probability more than about 20% away from where it started, then reuse each batch of experience for several passes. It is simple, robust, and became the default policy-gradient algorithm, including for training language models from human feedback.

The paper in four sentences

Policy-gradient learning makes good actions more likely and bad ones less likely, but large or repeated updates on the same batch of experience can wreck the policy.

TRPO fixed this with a hard limit on how much the behaviour may change, solved with heavy second-order maths. PPO gets most of the same effect with a one-line change to the objective: for each action, compare its new probability with its old one, and stop rewarding the update once that ratio leaves the range 0.8 to 1.2.

Because the clipped objective stops pulling once the policy has moved far enough, the same batch can safely be used for several passes of ordinary gradient descent, which makes learning much more data-efficient than plain policy gradients.

On simulated robots and Atari, PPO matched or beat the alternatives while being far simpler to implement, and it went on to become the workhorse of reinforcement learning from human feedback.

Idea · §1–2

The problem PPO solves

A policy is a network that, given a situation, outputs the probability of each action. A policy gradient collects some experience, works out which actions did better than expected (their advantage is positive) and which did worse, and shifts probability accordingly.

Collecting experience is expensive (you have to run the robot or play the game), so you would like to squeeze several learning steps out of each batch. But the batch only tells you about behaviour close to the policy that collected it. Keep optimising on it and the policy drifts far away, chasing patterns that held only for the old behaviour, and performance can collapse.

The earlier fix, TRPO, limits each update so the new policy stays within a small "distance" of the old one. It works, but it needs a special second-order optimisation step that is fiddly to implement and does not mix well with common network designs. PPO's authors, from the same group, wanted the same stability using only the plain gradient steps every deep learning library already provides.

Everyday pictureYou get feedback on a draft essay and revise it. One careful revision helps. Revising the same paragraph twenty times against the same single comment, without fresh feedback, tends to overcorrect. PPO lets you revise several times, but stops counting any change that has already gone well past what the feedback was about.

TakeawayThe goal: reuse each batch of experience for several updates without letting the policy run away from the data it was based on, using only ordinary gradient steps.

Idea · §3 · Eq. 7 · Fig. 1

The clip: stop rewarding change after 20%

For each action in the batch, PPO looks at a single number, the probability ratio: the action's probability under the policy being trained divided by its probability under the policy that collected the data. A ratio of 1 means "unchanged", 1.5 means "now 50% more likely", 0.7 means "30% less likely".

The ordinary objective is "ratio × advantage": for a good action, raising its ratio always looks better, without limit. PPO clips the ratio to a band, by default 0.8 to 1.2 (ε = 0.2). For a good action, once the ratio passes 1.2 the objective goes flat, so there is no more reward for pushing further. For a bad action, once the ratio drops below 0.8 it goes flat too. And PPO always takes the more pessimistic of the clipped and unclipped versions, so the clip can only ever remove incentives, never add them.

A re-drawing of the paper's Fig. 1 for one action with advantage +1 or −1. Dashed: the unclipped objective r × A. Amber: PPO's objective min(r × A, clip(r, 1−ε, 1+ε) × A). Where the amber line is flat, the gradient is zero, so this action stops pushing the update. Note the asymmetry: if a bad action has become more likely (r > 1), the objective still pulls it back down.

TakeawayPPO's objective gives credit for moving an action's probability only up to about ±20% from where it started; beyond that, the incentive vanishes.

Check yourself
A good action's ratio is already 1.4. Does PPO keep pushing it up?
No. With ε = 0.2 its term is clipped at 1.2 × A, so its gradient is zero and it no longer drives the update.
Why take the minimum of the clipped and unclipped terms rather than just the clipped one?
So the objective is pessimistic: if an update has accidentally made things worse for an action (a bad action became more likely), that mistake still shows up in full and gets corrected.

Idea · §3 · §5 · Algorithm 1

Several passes over each batch, safely

The payoff of clipping is reuse. PPO collects a batch of experience, then runs several epochs (passes) of gradient steps over it, between 3 and 15 in the paper's experiments, before collecting fresh data. Without the clip, more passes over the same batch keep pushing the same actions further and further, and the policy can lock onto a mediocre choice. With the clip, actions that have already moved enough drop out of the update.

Try it on a three-slot-machine toy: the machines pay 1.0, 0.7 and 0.2 on average, with plenty of noise. Each round the agent plays 16 times, then does several passes over those 16 results.

A toy, not the paper's experiment. Both agents use the same batches, the same advantage (reward minus batch average) and the same step size; the only difference is the objective, ratio × advantage (left) or PPO's clipped version with ε = 0.2 (right). Green = best machine. With 1 pass they are identical; with 10 or more passes the unclipped agent often commits to the 0.7 machine.

TakeawayClipping is what makes it safe to squeeze several updates out of every batch of experience.

Idea · §5 · Eq. 9

The rest of the recipe: a critic and a curiosity bonus

To know whether an action was "better than expected", you need an expectation. PPO trains a second output, the value (or "critic"), which predicts the future reward from each situation. The advantage of an action is then roughly "what actually happened afterwards minus what the critic predicted". The paper uses a method called generalised advantage estimation, which blends short-term and long-term evidence with a knob λ.

The final objective adds three things: the clipped policy term, a penalty for the critic's prediction errors, and a small entropy bonus that rewards the policy for staying a bit random, so it keeps exploring. Data come from short stretches of play (128 steps from each of 8 parallel game copies in Atari; 2,048 steps in the robot tasks) before each update.

Everyday pictureA sports coach (the critic) predicts how a play will go. If the play turns out better than the coach predicted, the player does more of it; if worse, less. Meanwhile the coach updates their predictions, and everyone is encouraged to keep trying the occasional new move.

TakeawayPPO in practice is an actor–critic method: clipped policy updates, a learned value baseline for advantages, and an entropy bonus for exploration.

Idea · §6 · Tables 1–2

Results, and an unexpected second career

On seven simulated robot tasks, the clipped version with ε = 0.2 scored best among all the variants the authors tried, including penalty-based versions closer to TRPO. Removing the clip entirely, while still taking many passes over each batch, scored worse than a random policy on average. On 49 Atari games it beat two strong rivals, A2C and ACER, on learning speed; on final performance ACER won more games. It also learned a 3-D humanoid robot that runs towards moving targets and gets up after being knocked over.

PPO's real legacy came later. Its combination of simplicity and robustness made it the standard choice for the final stage of training chat models: InstructGPT used PPO to fine-tune a language model against a reward model trained on human preferences.

Later refinementFor language models, PPO's critic is a large extra network to train. GRPO (2024) drops it: it samples a group of answers to the same prompt and uses each answer's reward relative to the group average as its advantage, while keeping PPO's clipped ratio. Direct preference optimisation (DPO, 2023) skips reinforcement learning altogether for preference data.

TakeawayPPO was about as good as anything else on benchmarks and far simpler, which is why it became the default, from robots to RLHF.

Follow the idea

Built on: TRPO (2015)

Led to: InstructGPT (RLHF) (2022), GRPO (2024)