Policy Optimization

Topics: Reinforcement Learning

Concept

Consider a parametrized Policy $\pi_\theta$. The goal is to maximize the expected return
$J(\pi_\theta) = \underset{\tau \sim \pi_\theta}{E}[R(\tau)]$
The essence is that we would like to optimize the policy by using gradient ascent on the network parameters:
$\theta_{k+1} = \theta_k + \alpha \cdot \nabla_\theta J(\pi_\theta) \vert_{\theta_k}$
where:
$\nabla_\theta J(\pi_\theta) \vert_{\theta_k}$ is the Policy Gradient
$\alpha$ is the Learning Rate

In order to use the above equation, we need an expression for the policy gradient. The full derivation of the Simple Policy Gradient is found in Simple Policy Gradient Derivation.

Types of Policy Gradients

Building on the previous derivation, we can write the policy gradient in the following general form:
$\nabla_\theta J(\pi_\theta) = \underset{\tau \sim \pi_\theta}{E} \left[ \sum_{t=0}^T \nabla_\theta \; log(\pi_\theta(a_t \vert s_t)) \cdot \Phi_t \right]$
where $\Phi_t$ can be any of the following:

1. Simplest Policy Gradient
   $\Phi_t = R(\tau)$
2. Reward to-Go Policy Gradient:
   $\Phi_t = \sum_{t'=t}^T R(s_{t'},a_{t'},s_{t'+1})$
3. Reward to-Go + Baseline Policy Gradient:
   $\Phi_t = \sum_{t'=t}^T R(s_{t'},a_{t'},s_{t'+1}) - b(s_t)$
   Choices 1, 2 and 3 lead to the same expected value but different variances. There are different choices of $\Phi_t$ which use the action values instead of rewards:
4. On-Policy Action-Value Policy Gradient:
   $\Phi_t = Q^{\pi_\theta}(s_t,a_t)$
5. Advantage-Based Policy Gradient:
   $\Phi_t = A^{\pi_\theta}(s_t,a_t)$
   where $A^{\pi_\theta}(s_t,a_t) = Q^{\pi_\theta}(s_t,a_t) - V^{\pi_\theta}(s_t)$.
   $A$ is the Advantage Function of an action, which describes how better or worse the action is compared to other actions on average.
   Advantage-based policy gradient can be thought of as On-Policy Action-Value Policy Gradient with a baseline equal to $V^{\pi_\theta}(s_t)$.

References

1. https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html