REINFORCE policy gradient

upshard

\nabla_{\theta} J = \mathbb{E}_{\pi_\theta}\!\left[\nabla_{\theta} \log \pi_{\theta}(a \mid s)\, G_t\right]

Raises the log-probability of actions that led to high return and lowers it for the rest. It optimises the policy directly with no value model required, and its variance is high enough that a baseline is essentially mandatory.

Symbols

$J$: The expected return being maximised.
$\pi_{\theta}$: The policy, parameterised by theta.
$G_t$: The return that followed the action.

Save your progress

Your account

Delete your account?

REINFORCE policy gradient

Keyboard shortcuts