PPO clipped objective

upshard

L = \mathbb{E}\!\left[\min\!\left(\rho_t A_t, \; \mathrm{clip}(\rho_t, 1-\epsilon, 1+\epsilon) A_t\right)\right]

Limits how far one update may move the policy by clipping the probability ratio and taking the pessimistic branch. That single trick is why PPO trains stably enough to be the workhorse of RLHF.

Symbols

$\rho_t$: The ratio of new to old action probability.
$A_t$: The advantage estimate for that step.
$\epsilon$: The clip width, typically 0.2.

Save your progress

Your account

Delete your account?

PPO clipped objective

Keyboard shortcuts