Vanilla Policy Gradient

Topics: Reinforcement Learning

Overview

Vanilla Policy Gradient (VPG) is a model-free, policy optimization, algorithm that trains a stochastic policy in an on-policy way. It can be used in environments with discrete or continuous action spaces.

Its underlying principle is to push up the probabilities of actions that lead to a higher return and push down the probabilities of actions that lead to a lower return, until the optimal policy is reached.

VPG is the basic policy optimization algorithm which updates the parameters $\theta$ of the actor network $\pi_\theta$ using gradient ascent. It also estimates the on-policy value function $V^\pi(s)$, which is used in approximating the advantage, which is in turn used in estimating the policy gradient.

Pseudocode

Exploration vs. Exploitation

Since VPG is on-policy, it samples actions from its current policy. Therefore, during the course of training, the policy becomes less random, which reduces exploration and could lead the policy to get stuck in a local optimum.

References

1. https://spinningup.openai.com/en/latest/algorithms/vpg.html