Simple Policy Gradient Derivation

Topics: Reinforcement Learning

The Policy Gradient is $\nabla_\theta J(\pi_\theta) \vert_{\theta_k}$. The goal is to derive an expression for it which can be calculated from samples of the interaction of the agent with the environment. The derivation is divided into 2 parts:

• derive the analytical equation of the policy gradient. It turns out that it has an expectation form $E(X)$
• derive the equation that estimated the expectation from samples (mean of the possible values of a random variable)

Analytical Derivation

In the following derivation, a stochastic policy $\pi_\theta$ and a finite-horizon undiscounted return are considered.

1- Probability of a Trajectory

$P(\tau|\theta) = \rho_0(s_0) \prod_{t=0}^{T} P(s_{t+1} \vert s_t, a_t)\; \pi_\theta(a_t \vert s_t)$

2- Log-Derivative Trick

$\nabla_\theta P(\tau|\theta)=P(\tau|\theta) \cdot \nabla_\theta\;log[P(\tau|\theta)]$

3- Log-Probability of a Trajectory

$log[P(\tau|\theta)] = log(\rho_0(s_0)) + \sum_{t=0}^T \left[ log( P(s_{t+1} \vert s_t, a_t)) + log(\pi_\theta(a_t \vert s_t)) \right] \quad\quad\quad\;\;\; log(a \cdot b)=log(a)+log(b)$

4- Gradient of Log-Probability of a Trajectory

$$\begin{align*} & \nabla_\theta log[P(\tau|\theta)] = \nabla_\theta \rho_0(s_0) + \sum_{t=0}^T[\nabla_\theta log( P(s_{t+1} \vert s_t, a_t)) + \nabla_\theta log(\pi_\theta(a_t \vert s_t))] \\ \\ & \nabla_\theta log[P(\tau|\theta)] = \sum_{t=0}^T \nabla_\theta log(\pi_\theta(a_t \vert s_t)) \end{align*}$$

because the other 2 terms are not a function of $\theta$.

5- Derivation for Simple Policy Gradient

$$\begin{align*} & \nabla_\theta J(\pi_\theta) = \nabla_\theta \underset{\tau \sim \pi_\theta}{E}[R(\tau)] \\ \\ & \quad\quad\quad\;\;\; = \nabla_\theta \int_\tau P(\tau|\theta) \cdot R(\tau) \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\; expand\;expectation \\ \\ & \quad\quad\quad\;\;\; = \int_\tau \nabla_\theta P(\tau|\theta) \cdot R(\tau) \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\;\; bring\;\nabla_\theta \; inside \; \int \\ \\ & \quad\quad\quad\;\;\; = \int_\tau P(\tau|\theta) \cdot \nabla_\theta\;log[P(\tau|\theta)] \cdot R(\tau) \quad\quad\quad\quad\quad\;\;\; log-derivative \; trick\;(2) \\ \\ & \quad\quad\quad\;\;\; = \underset{\tau \sim \pi_\theta}{E}[\nabla_\theta\;log[P(\tau|\theta)] \cdot R(\tau)] \quad\quad\quad\quad\quad\quad\quad\quad\; back\;to\;expectation\;form \\ \\ & \nabla_\theta J(\pi_\theta) = \underset{\tau \sim \pi_\theta}{E} \left[ \sum_{t=0}^T \nabla_\theta \; log(\pi_\theta(a_t \vert s_t)) \cdot R(\tau) \right] \quad\quad\quad\quad\quad replace\;expression\;for\;grad-log-prob\;(4) \end{align*}$$

Estimation of Policy Gradient from Samples

Since $\nabla_\theta J(\pi_\theta)$ is an expectation, we can estimate it with a sample mean
$\overset{\wedge}{g}=\frac{1}{\vert D \vert} \sum_{\tau\in D} \sum_{t=0}^T \nabla_\theta log(\pi_\theta(a_t \vert s_t)) \cdot R(\tau)$
where $D$ is the number of trajectories collected as samples.

References

1. https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html