TRPO

Topics: Reinforcement Learning

Introduction

Trust Region Policy Optimization (TRPO) is an on-policy RL algorithm that can be used to train a stochastic policy with either discrete or continuous action spaces. The idea is that TRPO updates the policy by taking the largest possible step while satisfying a constraint between the old and new policy expressed in terms of a Kullback-Leibler Divergence. The reason for using a KL divergence constraint, is that, in regular policy gradient, the old and new policy are close in parameter space, but not necessarily in terms of the probability distributions. A small difference in parameter space might have a large effect on the probability distribution and consequently on performance. By using KL divergence, TRPO keeps the old and new distributions close and tends to improve performance quickly and monotonically (1)

Equations

The theoretical optimization problem that TRPO solves is the following:

$$\begin{align*} & \theta_{k+1} = \underset{\theta}{\text{arg max}} \; \underset{s,a \sim \pi_{\theta_k}}{E} \Bigg[ \frac{\pi_{\theta}(a \vert s)}{\pi_{\theta_k}(a\vert s)} \; A^{\theta_{k}}(s,a)\Bigg] \\ \\ & \text{s.t.} \;\; \underset{s \sim \pi_{\theta_k}}{E} \bigg[ D_{KL} \Big( \pi_{\theta}(\cdot \, \vert \, a) \, \vert \vert \, \pi_{\theta_k}(\cdot \, \vert \, a) \Big) \bigg] \leq \delta \end{align*}$$

where $A^{\theta_{k}}$ is an estimator of the advantage function $Q(s,a)-V(s)$

The intuition behind the term that in the maximization is as follows:

• if $A^{\theta_{k}} \gt 0$, then the action $a$ is on average better than $V(s)$. Therefore, $\pi_{\theta}(a \vert s)$ should be increased, which means maximizing the positive value in the parentheses.
• if $A^{\theta_{k}} \lt 0$, then the action $a$ is worse than the average $V(s)$. Therefore, $\pi_{\theta}(a \vert s)$ should be decreased, and the problem is a maximization of a negative value

Approximations

The theoretical optimization problem of TRPO is difficult to work with, so approximations are made to get answers quickly

TODO: Go into the details of the approximations from the paper and (1)

Pseudocode

References

1. https://spinningup.openai.com/en/latest/algorithms/trpo.html