PPO

Topics: Reinforcement Learning

Introduction

Proximal Policy Optimization (PPO) builds upon the same idea as TRPO: how can we take the biggest possible step in the policy while satisfying a divergence constraint so that the performance does not accidentally collapse?
Similarly to TRPO, it is an on-policy RL algorithm that can be used to train a stochastic policy with either discrete or continuous action spaces.
While TRPO solves the optimization problem with a complex second-order method, PPO is a first-order method that uses a few tricks, which makes it significantly simpler to implement.
There are 2 main variants of PPO: PPO-Penalty and PPO-Clip, which are discussed below.

Optimization Problem

Here is a reminder of the TRPO optimization problem:

$$\begin{align*} & \theta_{t+1} = \underset{\theta}{\text{arg max}} \; \hat{E}_t \Bigg[ \frac{\pi_{\theta}(a_t \vert s_t)}{\pi_{\theta_{old}}(a_t\vert s_t)} \; \hat{A}_t(s_t,a_t)\Bigg] \\ \\ & \text{s.t.} \;\; \underset{s \sim \pi_{\theta_k}}{E} \bigg[ D_{KL} \Big( \pi_{\theta}(\cdot \, \vert \, a) \, \vert \vert \, \pi_{\theta_k}(\cdot \, \vert \, a) \Big) \bigg] \leq \delta \end{align*}$$

PPO-Penalty

PPO-Penalty adds a penalty in the reward term instead of the constraint, turning the problem into an unconstrained optimization (2):

$$\underset{\theta}{max} \; \hat{E}_t\Bigg[ \frac{\pi_{\theta}(a_t \vert s_t)}{\pi_{\theta_{old}}(a_t\vert s_t)} \; \hat{A}(s,a) - \beta \cdot KL\big[ {\pi_{\theta_{old}}(\cdot\vert s_t)}, \, \pi_{\theta}(\cdot \vert s_t) \big] \Bigg]$$

The first term of the optimization is the same surrogate objective from the TRPO problem. The second term aims to minimize the KL divergence.
The hyperparameter $\beta$ controls the weight of KL vs. objective and needs to be tuned. However, it is difficult to choose a single value of $\beta$ that performs well across different problems or even within the same problem across different stages.
Therefore, the $\beta$ can be adapted to achieve some target value of KL divergence $d_{targ}$ each policy update:

$$d=\hat{E}_t \bigg[ KL\big[ {\pi_{\theta_{old}}(\cdot\vert s_t)}, \, \pi_{\theta}(\cdot \vert s_t) \big] \bigg]$$

• if $d \lt d_{targ} / 1.5$ then $\beta \leftarrow \beta / 2$
• if $d \lt d_{targ} \times 1.5$ then $\beta \leftarrow \beta \times 2$

The updated $\beta$ value is used for the next policy update.

PPO-Clip

L^{CPI}(\theta)=\hat{E}_t\Bigg[ \frac{\pi_{\theta}(a_t \vert s_t)}{\pi_{\theta_{old}}(a_t\vert s_t)} \; \hat{A}_t \Bigg] $$Without a constraint, maximizing $L^{CPI}(\theta)$ would lead to excessively large policy updates. So, the clipped objective is proposed as follows:

L^{CLIP}(\theta)=\hat{E}t\Bigg[ \text{min} \bigg( \frac{\pi{\theta}(a_t \vert s_t)}{\pi_{\theta_{old}}(a_t\vert s_t)} ; \hat{A}t, ; \text{clip} \Big( \frac{\pi{\theta}(a_t \vert s_t)}{\pi_{\theta_{old}}(a_t\vert s_t)}, , 1-\epsilon, , 1+\epsilon \Big) \hat{A}_t \bigg) \Bigg]

where $\epsilon$ is a hyperparameter. The clipping removes the possibility for moving $\frac{\pi_{\theta}(a_t \vert s_t)}{\pi_{\theta_{old}}(a_t\vert s_t)}$ outside of the interval $[1-\epsilon, \, 1+\epsilon]$. This puts a limit on how much $\pi_{\theta}(a_t \vert s_t)$ can change with respect to $\pi_{\theta_{old}}(a_t\vert s_t)$. ## Pseudocode  ## References 1. https://spinningup.openai.com/en/latest/algorithms/ppo.html 2. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” Aug. 28, 2017, _arXiv_: arXiv:1707.06347. doi: [10.48550/arXiv.1707.06347](https://doi.org/10.48550/arXiv.1707.06347).