Reward to-Go + Baseline Policy Gradient

Topics: Reinforcement Learning

Given the Expected Gradient Log-Probability Lemma:
$\underset{a_t \sim \pi_\theta}{E}[\nabla_\theta \; log(\pi_\theta(a_t \vert s_t))] = 0$

Then, for any term $b$ that only depends on state, we have:
$\underset{a_t \sim \pi_\theta}{E}[\nabla_\theta \; log(\pi_\theta(a_t \vert s_t)) \cdot b(s_t)] = 0$

Therefore, we can add/subtract $b$ in the Reward to-Go Policy Gradient without changing the value of the expectation:
$\nabla_\theta J(\pi_\theta) = \underset{\tau \sim \pi_\theta}{E} \left[ \sum_{t=0}^T \nabla_\theta \; log(\pi_\theta(a_t \vert s_t)) \cdot \left( \sum_{t'=t}^T R(s_{t'},a_{t'},s_{t'+1}) -b(s_t) \right) \right]$
where $b(s_t)$ is referred to as Baseline.

The most common choice of $b(s_t)$ is the on-policy value function $V^\pi(s_t)$. This reduces the variance in the sample estimate of policy gradient.

Intuitively, it allows to compare the earned rewards of the agent to the expected average value. If the agent gets the reward it expected, then it should feel neutral about it: $\sum R - b = 0$.

References

1. https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html