Diagonal Gaussian Policy

Topics: Reinforcement Learning

Introduction

A Diagonal Gaussian Policy is a type of Stochastic Policies that is used for continuous action spaces.
In continuous action spaces, the normal distribution is used in the policy.

1-dimensional Action

If the action is 1-dimensional, then we have a single normal distribution with mean $\mu$ and standard deviation $\sigma$.
The policy consists of a neural network that maps state to mean $\mu_\theta(s)$. Regarding the variance, we can have one of two options:

• a standalone parameter (not function) of log standard deviation $log\;\sigma$
• a neural network that maps state to log standard deviation $log\;\sigma_\theta(s)$. It may share layers with the mean network $\mu_\theta(s)$.

k-dimensional Action

If the action is k-dimensional, a Multivariate Gaussian Distribution is used. It is characterized by a mean vector $\mu$ and a covariance matrix $\Sigma$. A special case of the Multivariate Gaussian Distribution, called the Diagonal Gaussian Distribution, is usually used. It is distinguished by a diagonal covariance matrix $\Sigma$, which can be represented by a vector.
Similarly to the 1-D action case, the Diagonal Gaussian Policy has a neural network for mean $\mu_\theta$ and either a standalone parameter or neural network for the log standard deviation $log\;\sigma$.

N.B.: We use $log\;\sigma$ instead of $\sigma$ because $log\;\sigma\in[-\infty,+\infty]$ while $\sigma>0$. So, it is easier to train a neural network on $\log\;\sigma$ and we do not lose any information since we can convert $log\;\sigma$ to $\sigma$ via the exponential function.

Sampling

$a=\mu_\theta(a)+\sigma_\theta(s)\,\odot\,z$
where $z$ is a vector of noise from a Spherical Gaussian $z\sim \mathcal{N}(0,I)$ and $\odot$ is element-wise multiplication.

Log-likelihood

$log\;\pi_\theta(a|s)=-\frac{1}{2}(\sum^{k}_{i=1}{(\frac{(a_i-\mu_i)^2}{\sigma_i^2}+2\:log\;\sigma_i)+k\:log\;2\pi})$
where $k$ is the dimension of the action

References

1. https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#