The paper in four sentences
Policy-gradient learning makes good actions more likely and bad ones less likely, but large or repeated updates on the same batch of experience can wreck the policy.
TRPO fixed this with a hard limit on how much the behaviour may change, solved with heavy second-order maths. PPO gets most of the same effect with a one-line change to the objective: for each action, compare its new probability with its old one, and stop rewarding the update once that ratio leaves the range 0.8 to 1.2.
Because the clipped objective stops pulling once the policy has moved far enough, the same batch can safely be used for several passes of ordinary gradient descent, which makes learning much more data-efficient than plain policy gradients.
On simulated robots and Atari, PPO matched or beat the alternatives while being far simpler to implement, and it went on to become the workhorse of reinforcement learning from human feedback.
Idea · §1–2
The problem PPO solves
A policy is a network that, given a situation, outputs the probability of each action. A policy gradient collects some experience, works out which actions did better than expected (their advantage is positive) and which did worse, and shifts probability accordingly.
Collecting experience is expensive (you have to run the robot or play the game), so you would like to squeeze several learning steps out of each batch. But the batch only tells you about behaviour close to the policy that collected it. Keep optimising on it and the policy drifts far away, chasing patterns that held only for the old behaviour, and performance can collapse.
The earlier fix, TRPO, limits each update so the new policy stays within a small "distance" of the old one. It works, but it needs a special second-order optimisation step that is fiddly to implement and does not mix well with common network designs. PPO's authors, from the same group, wanted the same stability using only the plain gradient steps every deep learning library already provides.
TakeawayThe goal: reuse each batch of experience for several updates without letting the policy run away from the data it was based on, using only ordinary gradient steps.
Idea · §3 · Eq. 7 · Fig. 1
The clip: stop rewarding change after 20%
For each action in the batch, PPO looks at a single number, the probability ratio: the action's probability under the policy being trained divided by its probability under the policy that collected the data. A ratio of 1 means "unchanged", 1.5 means "now 50% more likely", 0.7 means "30% less likely".
The ordinary objective is "ratio × advantage": for a good action, raising its ratio always looks better, without limit. PPO clips the ratio to a band, by default 0.8 to 1.2 (ε = 0.2). For a good action, once the ratio passes 1.2 the objective goes flat, so there is no more reward for pushing further. For a bad action, once the ratio drops below 0.8 it goes flat too. And PPO always takes the more pessimistic of the clipped and unclipped versions, so the clip can only ever remove incentives, never add them.
TakeawayPPO's objective gives credit for moving an action's probability only up to about ±20% from where it started; beyond that, the incentive vanishes.
Check yourself
- A good action's ratio is already 1.4. Does PPO keep pushing it up?
- No. With ε = 0.2 its term is clipped at 1.2 × A, so its gradient is zero and it no longer drives the update.
- Why take the minimum of the clipped and unclipped terms rather than just the clipped one?
- So the objective is pessimistic: if an update has accidentally made things worse for an action (a bad action became more likely), that mistake still shows up in full and gets corrected.
Idea · §3 · §5 · Algorithm 1
Several passes over each batch, safely
The payoff of clipping is reuse. PPO collects a batch of experience, then runs several epochs (passes) of gradient steps over it, between 3 and 15 in the paper's experiments, before collecting fresh data. Without the clip, more passes over the same batch keep pushing the same actions further and further, and the policy can lock onto a mediocre choice. With the clip, actions that have already moved enough drop out of the update.
Try it on a three-slot-machine toy: the machines pay 1.0, 0.7 and 0.2 on average, with plenty of noise. Each round the agent plays 16 times, then does several passes over those 16 results.
TakeawayClipping is what makes it safe to squeeze several updates out of every batch of experience.
Idea · §5 · Eq. 9
The rest of the recipe: a critic and a curiosity bonus
To know whether an action was "better than expected", you need an expectation. PPO trains a second output, the value (or "critic"), which predicts the future reward from each situation. The advantage of an action is then roughly "what actually happened afterwards minus what the critic predicted". The paper uses a method called generalised advantage estimation, which blends short-term and long-term evidence with a knob λ.
The final objective adds three things: the clipped policy term, a penalty for the critic's prediction errors, and a small entropy bonus that rewards the policy for staying a bit random, so it keeps exploring. Data come from short stretches of play (128 steps from each of 8 parallel game copies in Atari; 2,048 steps in the robot tasks) before each update.
TakeawayPPO in practice is an actor–critic method: clipped policy updates, a learned value baseline for advantages, and an entropy bonus for exploration.
Idea · §6 · Tables 1–2
Results, and an unexpected second career
On seven simulated robot tasks, the clipped version with ε = 0.2 scored best among all the variants the authors tried, including penalty-based versions closer to TRPO. Removing the clip entirely, while still taking many passes over each batch, scored worse than a random policy on average. On 49 Atari games it beat two strong rivals, A2C and ACER, on learning speed; on final performance ACER won more games. It also learned a 3-D humanoid robot that runs towards moving targets and gets up after being knocked over.
PPO's real legacy came later. Its combination of simplicity and robustness made it the standard choice for the final stage of training chat models: InstructGPT used PPO to fine-tune a language model against a reward model trained on human preferences.
TakeawayPPO was about as good as anything else on benchmarks and far simpler, which is why it became the default, from robots to RLHF.
Abstract, restated
A new family of policy-gradient methods that alternate between sampling data through interaction with the environment and optimising a "surrogate" objective with stochastic gradient ascent. Unlike standard policy gradients, which do one update per sample, the objective allows multiple epochs of minibatch updates. Proximal Policy Optimization keeps some of the benefits of TRPO but is much simpler to implement, more general and has better sample complexity empirically. On simulated robotic locomotion and Atari it outperforms other online policy-gradient methods and strikes a favourable balance of sample complexity, simplicity and wall-time. arXiv:1707.06347 (2017).
Section · §2 · Eqs. 1–5
Background: policy gradients and TRPO
The standard estimator and the loss whose gradient produces it:
Êt is an empirical average over a finite batch. Doing multiple optimisation steps on LPG with the same trajectory "is not well-justified" and empirically leads to destructively large updates. TRPO instead solves
by conjugate gradient after linear and quadratic approximations. The theory behind TRPO actually suggests a penalty, Êt[ratio · Ât − β KL] (Eq. 5), but a single β that works across problems, or even across one run, is hard to find; that motivates PPO's first-order alternatives.
Section · §3 · Eqs. 6–7 · Figs. 1–2
The clipped surrogate objective
With rt(θ) = πθ(at|st) / πθold(at|st), so r(θold) = 1, the conservative-policy-iteration surrogate is LCPI(θ) = Êt[rt(θ) Ât] (Eq. 6). Unconstrained, it invites excessively large updates. PPO's main objective is
with ε a hyperparameter, e.g. ε = 0.2. The second term removes the incentive for moving rt outside [1 − ε, 1 + ε]; the minimum makes the result a lower (pessimistic) bound on the unclipped objective. The clip is active only when it would make the objective larger: for  > 0 when r > 1 + ε, for  < 0 when r < 1 − ε. Changes in r that make the objective worse are always included. To first order around θold, LCLIP equals LCPI.
Fig. 2 interpolates along the update direction from one PPO iteration on Hopper-v1: LCLIP lies below LCPI and is maximal at the end of the update, where the KL is about 0.02, so it behaves like LCPI with a penalty for large updates.
A caveat noticed later: clipping removes the gradient from samples whose ratio is out of range but does not bound the ratio itself. With many epochs and large learning rates the ratio, and the KL, can still drift beyond 1 ± ε, which is why many implementations add early stopping on KL.
Section · §4 · Eq. 8
The alternative: adaptive KL penalty
After several epochs of minibatch SGD on Eq. 8, compute d = Êt[KL]. If d < dtarg / 1.5, halve β; if d > 1.5 dtarg, double it. The constants 1.5 and 2 are heuristic, and the algorithm is insensitive to them and to β's initial value. The authors found it performed worse than clipping but kept it as an important baseline. Heess et al. (2017) used this variant for their 3-D locomotion results.
Section · §5 · Eqs. 9–12 · Algorithm 1
The full algorithm
With a network sharing parameters between policy and value function, the combined objective (maximised) is
LVF = (Vθ(st) − Vttarg)2, S is an entropy bonus. Advantages use truncated generalised advantage estimation over a segment of length T:
With λ = 1 this reduces to the finite-horizon estimator of Eq. 10 (Mnih et al., 2016). (The arXiv text prints the last exponent as T − t + 1; the sum written out here is the intended one, ending at δT−1.)
Algorithm 1: PPO, actor-critic style
for iteration = 1, 2, …:
for actor = 1 … N:
run π_θold in the environment for T timesteps
compute advantage estimates Â₁ … Â_T
optimise surrogate L wrt θ, with K epochs and minibatch size M ≤ NT
θold ← θ
| Hyperparameter | MuJoCo (Table 3) | Atari (Table 5) |
|---|---|---|
| Horizon T | 2048 | 128 |
| Adam step size | 3 × 10⁻⁴ | 2.5 × 10⁻⁴ × α |
| Epochs K | 10 | 3 |
| Minibatch | 64 | 32 × 8 |
| γ, λ | 0.99, 0.95 | 0.99, 0.95 |
| Actors N | – | 8 |
| Clip ε | 0.2 | 0.1 × α |
| c1, c2 | separate networks, no entropy | 1, 0.01 |
α is annealed linearly from 1 to 0 over training. MuJoCo runs use 1 million timesteps; policy and value are separate networks with two hidden layers of 64 tanh units; the policy outputs the mean of a Gaussian with variable standard deviations.
Section · §6 · Tables 1–2 · Figs. 3–6
Experiments
Surrogate comparison (§6.1, Table 1). Seven MuJoCo tasks (OpenAI Gym), three seeds each, one million timesteps; the score of the last 100 episodes is normalised so a random policy is 0 and the best result is 1, then averaged over 21 runs.
Comparison with other algorithms (§6.2, Fig. 3). On the same seven tasks, PPO (clip) is compared with TRPO, CEM, vanilla policy gradient with adaptive step size, A2C and A2C with trust region; it outperforms the previous methods on almost all of the environments.
Humanoid showcase (§6.3, Figs. 4–5). Roboschool humanoid running, steering towards moving flags, and getting up after being hit by cubes, with 32 or 128 parallel actors and minibatches of 4,096.
Atari (§6.4, Table 2). 49 games, 40M frames, three seeds, compared with tuned A2C and ACER. Games "won" by each algorithm:
| Scoring metric | A2C | ACER | PPO | Tie |
|---|---|---|---|---|
| Average episode reward over all of training (learning speed) | 1 | 18 | 30 | 0 |
| Average episode reward over last 100 episodes (final performance) | 1 | 28 | 19 | 1 |
Check yourself
- Why does "no clipping or penalty" score below zero in Table 1?
- With multiple epochs of SGD per batch, the unconstrained LCPI drives very large policy updates; on some tasks (the paper mentions HalfCheetah) the policy collapses to worse than random.
- PPO wins more Atari games on metric (1) but fewer on (2). What does that say?
- PPO learns faster (higher reward averaged over training), but ACER, which uses experience replay, often reaches a higher final score.
Section · Lineage
Limitations and later refinements
- No guarantee. Unlike TRPO's theory, LCLIP has no monotonic-improvement proof, and clipping does not strictly enforce a trust region (Wang et al., 2019, "Truly PPO"; Ilyas et al., 2020).
- Implementation sensitivity. Engstrom et al. (2020) showed that code-level choices such as value clipping, reward and observation normalisation, orthogonal initialisation and learning-rate annealing account for much of PPO's advantage over TRPO; Huang et al.'s "37 implementation details" (2022) catalogues them.
- On-policy sample cost. Data are discarded after K epochs, so replay-based off-policy methods (SAC, TD3) are often more sample-efficient in continuous control.
Reference
Glossary
- rt(θ)
- Probability ratio πθ(at|st) / πθold(at|st).
- ε
- Clip range; ratios outside [1 − ε, 1 + ε] get no further credit.
- Advantage Â
- Estimate of how much better an action did than the critic expected.
- GAE(λ)
- Generalised advantage estimation: exponentially weighted sum of TD errors.
- Epoch
- One full pass of minibatch SGD over the current batch.
- Entropy bonus
- Reward for keeping the action distribution spread out, encouraging exploration.
- A2C / ACER
- Synchronous advantage actor-critic; actor-critic with experience replay.
- RLHF
- Reinforcement learning from human feedback: optimise a policy against a reward model trained on human preferences.