Expected Gradient Log-Probability Lemma

Topics: Reinforcement Learning

Suppose $P_\theta$ is a parametrized probability distribution over random variable $x$, for example a stochastic policy, which is a parametrized probability distribution over actions.
Then
$\underset{x \sim P_\theta}{E}[\nabla_\theta \; P_\theta(x)] = 0$

Proof

$$\begin{align*} & \int_x P_\theta(x) = 1 \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad probability\;distributions\;are\;normalized \\ \\ & \nabla_\theta \; \int_x P_\theta(x) = \nabla_\theta \; 1 = 0 \quad\quad\quad\quad\quad\quad gradient\;both\;sides \\ \\ & \int_x \nabla_\theta \; P_\theta(x) = 0 \\ \\ & \int_x P_\theta(x) \cdot \nabla_\theta \; log(P(x)) = 0 \quad\quad\quad\quad\;\; log-derivative\;trick \\ \\ & \underset{x \sim P_\theta}{E}[\nabla_\theta \; log(P(x))] = 0 \quad\quad\quad\quad\quad\quad\;\;\; back\;to\;expectation\;form \end{align*}$$

References

1. https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html