PPO

From TRPO to PPO#

TRPO (Schulman et al., 2015) maximises a surrogate objective subject to a hard KL constraint:

$$ \max_\theta \; \mathbb{E}t!\left[ \frac{\pi\theta(a_t \mid s_t)}{\pi_{\theta_{\text{old}}}(a_t \mid s_t)} \hat{A}_t \right] \quad \text{subject to} \quad \mathbb{E}t!\left[ \mathrm{KL}!\left( \pi{\theta_{\text{old}}}(\cdot \mid s_t) \,|\, \pi_\theta(\cdot \mid s_t) \right) \right] \le \delta $$

The motivation is a monotonic improvement guarantee (Kakade and Langford, 2002): the true return satisfies $J(\pi') \ge L_{\pi}(\pi') - C \cdot \max_s \mathrm{KL}(\pi | \pi')$ with $C = \frac{4\epsilon\gamma}{(1-\gamma)^2}$ and $\epsilon = \max_{s,a}|A^\pi(s,a)|$. Optimising the right-hand side never decreases $J$. In practice the theoretical $C$ forces steps so small that TRPO uses $\delta$ as a tuned hyperparameter instead, discarding the guarantee it was derived from.

TRPO solves the constrained problem with a conjugate-gradient step against a Fisher-vector product, plus a backtracking line search. It works, and it is awkward: second-order, hard to combine with parameter sharing, and unpleasant with recurrent networks.

The clipped surrogate#

PPO (Schulman et al., 2017) replaces the constraint with a clip. With $r_t(\theta) = \frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}$:

$$ L^{\text{CLIP}}(\theta) = \mathbb{E}_t\left[ \min\Big( r_t(\theta)\hat{A}_t, \; \mathrm{clip}\big(r_t(\theta), 1-\epsilon, 1+\epsilon\big)\hat{A}_t \Big) \right] $$

The $\min$ makes $L^{\text{CLIP}}$ a lower bound on the unclipped surrogate. Gradient behaviour, stated exactly:

The full objective adds a value loss and an entropy bonus: $L = L^{\text{CLIP}} - c_1 (V_\phi(s_t) - \hat{R}_t)^2 + c_2 \mathcal{H}\pi_\theta$, with $c_1 \approx 0.5$ and $c_2 \approx 0.01$.

PPO has no monotonic improvement guarantee. The clip bounds the per-sample likelihood ratio, not the expected KL, and Engstrom et al. (2020) exhibit runs where the mean KL substantially exceeds what the clip suggests. It is a heuristic that works well, which is a different claim from a theorem.

The implementation details#

Engstrom et al. (2020), Implementation Matters in Deep Policy Gradients, and Andrychowicz et al. (2020), What Matters in On-Policy Reinforcement Learning?, both ran large ablations. Their combined findings are uncomfortable and worth taking seriously.

The choices that mattered most, several of which appear nowhere in the PPO paper:

• Observation normalisation with a running mean and variance. Large effect on continuous control.
• Reward scaling by a running estimate of the return's standard deviation.
• Advantage normalisation per minibatch.
• Orthogonal initialisation with a policy output layer scaled by $0.01$, so the initial policy is near-uniform.
• Learning rate annealing to zero over training.
• Gradient clipping at global norm 0.5.
• Value function loss clipping — contested; several studies find it neutral or harmful.
• Adam epsilon at $10^{-5}$ rather than the default $10^{-8}$.

Engstrom et al.'s sharpest result: TRPO with PPO's code-level optimisations matches PPO. Much of the reported gap between the two algorithms came from the implementation, not the objective.

The practical consequence is that a from-scratch PPO that underperforms a library's is usually missing details, not misunderstanding the maths. Huang et al. (2022), The 37 Implementation Details of Proximal Policy Optimization, is the reference to work through.

Variants#

• PPO-Penalty. Adds $-\beta \,\mathrm{KL}$ to the objective with $\beta$ adapted to hit a target KL. Used in the original RLHF work, where an explicit KL budget against the pretrained model is the point rather than an implementation detail.
• Early stopping on KL. Abandon remaining epochs once the approximate KL exceeds roughly $1.5\times$ the target. Cheap and effective.
• Phasic Policy Gradient (Cobbe et al., 2021). Separates policy and value optimisation into distinct phases, letting the value function take many more passes than the policy safely.
• DAPO, GRPO and relatives (2024-2025). Group-relative variants that drop the learned critic and estimate the advantage from a group of sampled responses to the same prompt. Now common in LLM reasoning training, where a value network over token sequences is expensive and poorly conditioned.

References#

• Schulman et al. (2015), Trust Region Policy Optimization — arxiv.org/abs/1502.05477.
• Schulman et al. (2017), Proximal Policy Optimization Algorithms — arxiv.org/abs/1707.06347.
• Engstrom et al. (2020), Implementation Matters in Deep Policy Gradients — arxiv.org/abs/2005.12729.
• Andrychowicz et al. (2020), What Matters in On-Policy Reinforcement Learning? — arxiv.org/abs/2006.05990.
• Huang et al. (2022), The 37 Implementation Details of Proximal Policy Optimization.
• Cobbe et al. (2021), Phasic Policy Gradient — arxiv.org/abs/2009.04416.

What to learn next#

• RLHF — PPO doing the job it is most widely used for today.
• Reward shaping — because a stable algorithm on a wrong reward is still wrong.
• Actor-critic methods — the foundation PPO is built on.