Log-Derivative Trick

Topics: Reinforcement Learning

This note is the proof of the Log-Derivative trick used in policy gradients in RL:
$\nabla_\theta P(\tau|\theta)=P(\tau|\theta)\;.\;\nabla_\theta\;log[P(\tau|\theta)]$

Proof

$\tau$ : trajectory of states and actions
$\theta$ : parameters of the actor network

Let
$y=P(\tau|\theta)$
$z=log\;[P(\tau|\theta)]$

Then

$$\begin{align*} & \frac{d\;log[P(\tau|\theta)]}{d\theta}=\frac{dz}{d\theta}=\frac{dz}{dy}\cdot \frac{dy}{d\theta} \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad chain \; rule \\ \\ & \frac{d\;log[P(\tau|\theta)]}{d\theta}=\frac{d\;log[P(\tau|\theta)]}{d(P(\tau|\theta))}\cdot\frac{d(P(\tau|\theta))}{d\theta} \\ \\ & \frac{d\;log[P(\tau|\theta)]}{d\theta}=\frac{d\;log[P(\tau|\theta)]}{d(P(\tau|\theta))}\cdot\frac{d(P(\tau|\theta))}{d\theta} \\ \\ & \frac{d\;log[P(\tau|\theta)]}{d\theta}=\frac{1}{P(\tau|\theta)}\cdot \frac{d(P(\tau|\theta))}{d\theta} \quad\quad\quad\quad\quad\quad\quad\quad\frac{d}{dx}log(x)=\frac{1}{x} \\ \\ & \nabla_\theta\;log[P(\tau|\theta)] = \frac{1}{P(\tau|\theta)} \cdot \nabla_\theta P(\tau|\theta) \\ \\ & \nabla_\theta P(\tau|\theta)=P(\tau|\theta)\;.\;\nabla_\theta\;log[P(\tau|\theta)] \end{align*}$$

References

1. https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html
2. https://davidmeyer.github.io/ml/log_derivative_trick.pdf