GRPO

The objective#

GRPO (Shao et al., 2024, DeepSeekMath) samples a group ${o_1,\dots,o_G}$ from $\pi_{\theta_{\text{old}}}(\cdot\mid q)$ and optimises

$$ \mathcal{J}{\text{GRPO}}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min!\Big(\rho_{i,t}\hat A_{i,t},\ \mathrm{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\hat A_{i,t}\Big) - \beta\, \mathbb{D}{\mathrm{KL}}!\left[\pi\theta | \pi_{\text{ref}}\right]\right] $$

with importance ratio $\rho_{i,t} = \dfrac{\pi_\theta(o_{i,t}\mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q, o_{i,<t})}$ and the group-relative advantage

$$ \hat A_{i,t} = \frac{r_i - \mathrm{mean}({r_1,\dots,r_G})}{\mathrm{std}({r_1,\dots,r_G})} $$

Three structural points.

The advantage is constant across $t$. Every token in a completion receives the same advantage. There is no per-token credit assignment, and no GAE. This is what removes the value network.

The baseline is a Monte Carlo estimate of $V(q)$. PPO learns $V$; GRPO estimates it from $G$ samples at each step. The variance of that estimate is $O(1/G)$, which is why group size is the algorithm's most important hyperparameter.

The KL term is in the loss, not the reward. GRPO uses the k3 estimator directly in the objective, unlike PPO-style RLHF which folds a per-token KL into the reward signal.

Where GRPO is biased, and the corrections#

Liu et al., 2025 (Understanding R1-Zero-Like Training: A Critical Perspective, "Dr. GRPO") identify two biases in the formula above.

1. Length bias, from the $1/|o_i|$ term. Dividing by response length means each token of a long wrong answer is penalised less than each token of a short wrong answer. The optimisation therefore prefers longer wrong answers — which explains part of the response-length growth attributed to "more reasoning".
2. Difficulty bias, from dividing by $\mathrm{std}(r)$. Questions where the group's rewards happen to have low variance get their advantages inflated, over-weighting questions that are nearly always right or nearly always wrong.

Dr. GRPO removes both: drop the $1/|o_i|$ factor (use a constant $L$ instead) and drop the standard-deviation division. TRL exposes this as loss_type="dr_grpo" and scale_rewards=False.

DAPO (Yu et al., 2025, Decoupled Clip and Dynamic sAmpling Policy Optimization) makes four changes, all of which are now common practice:

• Clip-higher. Decouple $\epsilon_{\text{low}}$ and $\epsilon_{\text{high}}$, raising the upper bound. The symmetric clip caps how much probability a low-probability token can gain, which drives entropy collapse; a larger upper bound preserves exploration.
• Dynamic sampling. Oversample and discard groups where accuracy is exactly 0 or exactly 1. This directly targets the dead groups column in the code above — those groups contribute nothing but still consume batch capacity and inflate gradient variance.
• Token-level policy-gradient loss. Average over all tokens in the batch rather than averaging per-sequence means, so long sequences carry proportionate weight.
• Overlong reward shaping. A soft length penalty starting at one threshold and strong enough past a second threshold to cancel a correct answer's reward.

GSPO (Zheng et al., 2025, Qwen) argues the importance ratio should be defined at the sequence level, since the reward is a sequence-level quantity, and reports better stability for mixture-of-experts models where token-level ratios are especially noisy.

What RL on verifiable rewards actually adds#

This is contested and worth stating carefully.

DeepSeek-R1 (DeepSeek-AI, 2025) reported R1-Zero — GRPO applied directly to a base model with rule-based rewards, no SFT — reaching 71.0% on AIME 2024 (86.7% with majority voting), up from 15.6%. It also reported the emergence of longer chains of thought and self-correction as training proceeded. The released R1 added a small cold-start SFT stage before RL, because R1-Zero's outputs suffered from poor readability and language mixing.

The critical question is whether RL teaches new capability or sharpens existing capability.

• Yue et al., 2025 (Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?) compare pass@$k$ at large $k$ and find RL-trained models do not exceed their base models — RL concentrates probability on solutions the base model could already sample, improving pass@1 while sometimes reducing pass@256.
• The counter-position is that pass@$k$ at large $k$ is not the quantity anyone deploys, and that reliable pass@1 is the capability.

Both readings are consistent with the toy result above: nothing was learned in the first 100 steps because nothing was being sampled to reinforce.

Cost profile#

GRPO's real-world speed advantage is smaller than the memory saving suggests, because generation dominates the step time in both. Its decisive advantage is that it removes an entire model and its hyperparameters from a notoriously fragile pipeline.

Papers#

• Schulman et al., Proximal Policy Optimization Algorithms, 2017 — arxiv.org/abs/1707.06347
• Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024 — arxiv.org/abs/2402.03300 (introduces GRPO)
• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025 — arxiv.org/abs/2501.12948
• Liu et al., Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO), 2025 — arxiv.org/abs/2503.20783
• Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale, 2025 — arxiv.org/abs/2503.14476
• Yue et al., Does RL Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, 2025 — arxiv.org/abs/2504.13837
• Zheng et al., Group Sequence Policy Optimization (GSPO), 2025 — arxiv.org/abs/2507.18071
• TRL GRPO Trainer documentation — huggingface.co/docs/trl/grpo_trainer

What to learn next#

• RL with verifiable rewards — where the reward function comes from.
• Thinking tokens and reasoning models — what long GRPO runs produce.
• PPO — the algorithm GRPO simplifies.