Reward to-Go Policy Gradient

Topics: Reinforcement Learning

The problem with the simple policy gradient is that the log-probability at every time step $t$ is getting multiplied by the sum of all the rewards of the trajectory $R(\tau)$.
This does not make sense because the rewards obtained before an action was taken do not relate to how good or bad that action was. In other words, rewards are the consequences of actions and only rewards that were obtained after an action was taken should be considered.

Hence, the Reward to-Go Policy Gradient is formulated as follows:
$\nabla_\theta J(\pi_\theta) = \underset{\tau \sim \pi_\theta}{E} \left[ \sum_{t=0}^T \nabla_\theta \; log(\pi_\theta(a_t \vert s_t)) \cdot \sum_{t'=t}^T R(s_{t'},a_{t'},s_{t'+1}) \right]$
By replacing $R(\tau)$ with the reward to-go, we reduce the number of samples needed to get low variance for the policy gradient estimate.

References

1. https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html