Advantage — reward relative to a baseline
How much better a response scored than a typical response, R − b. Policy gradients weighted by the advantage are far less noisy than those weighted by the raw reward.
PPO gets the baseline from a value model; GRPO from the average of a group of sampled responses (sections 21.2, 21.3 and 22.3).
Related terms
- Policy — a rule for choosing actionsIn reinforcement learning, a probability distribution over actions given the current state. A language model is a policy over next tokens.
- Reward — a score for an outcomeA number saying how good an outcome was. For a chat response it comes from a learned reward model; for maths or code it can come from a checker.