DDPG

Topics: Reinforcement Learning

Introduction

Deep Deterministic Policy Gradient (DDPG) is a model-free, actor-critic, off-policy, RL algorithm that trains a deterministic policy targeted at environments with continuous action spaces.
It learns both a Q-function and a policy (actor-critic): it uses off-policy data and the Bellman Equation to learn the Q-values, and uses the Q-values to learn the policy.
DDPG can be thought of as Deep Q-Learning (DQN) for continuous action spaces.

Q-Learning Part of DDPG

Starting from the Bellman equation:
$Q(s,a) = r + \gamma \cdot \underset{a'}{max} \; Q(s',a')$
In the case of discrete action spaces, $\underset{a'}{max} \; Q(s',a')$ is very easy to compute. But in the case of continuous action spaces, like DDPG, we cannot do that, and the optimization problem of finding the maximum Q-value and optimal action is too expensive to run every time the agent needs to make a decision.
For continuous action spaces, we approximate it with:
$\underset{a'}{max} \; Q(s',a') = Q(s', \mu(s'))$
where $\mu$ is the policy.

Then, the Bellman equation becomes:
$Q(s,a) = r + \gamma \cdot Q(s',\mu(s'))$

Note that $\mu$ is the actor neural network with parameters $\theta$. Its input is the state $s$ and its output is the action $a$. $Q$ is the critic neural network with parameters $\phi$. Its inputs are the state $s$ and action $a$, and its output is the Q-value $Q(s,a)$.

So, to satisfy the Bellman equation, we can update the parameters $\phi$ by minimizing the Mean Square Bellman Error (MSBE):
$MSBE=\left[ Q_\phi(s,a) - \left( r + \gamma \cdot Q_\phi(s', \mu(s')) \right) \right] ^2$
A problem that arises from this formulation is that the network that we want to update, $Q_\phi$, is the same network used on the right side of the Bellman equation. It is as if you are tracking a target, but every time the you take a step closer, it moves a step further from you. This makes learning unstable.

Therefore, the idea of target networks was introduced, which is a delayed version of the network being trained. The parameters of the target networks get updated very slowly, which makes the target stable and hence the training stable.

The above image shows how a stored sample is used in training. The training of the thus consists of minimizing the following loss function:
$L(\phi,D)=\underset{(s,a,r,s') \sim D}{E} \left[ \left( Q_\phi(s,a) - \left( r + \gamma \cdot (1-d) \cdot Q_{\phi_{target}}(s', \mu_{\theta_{target}}(s')) \right) \right) ^2 \right]$

The parameters of the target networks are updated via Polyak averaging, which is also known as a soft update:

$$\begin{align*} & \theta_{target} = \rho \cdot \theta_{target} + (1 - \rho) \cdot \theta \\ \\ & \phi_{target} = \rho \cdot \phi_{target} + (1 - \rho) \cdot \phi \end{align*}$$

where $\rho$ is a constant close to $1$, for example $0.99$ or $0.999$.

The final point to discuss in this section is that, since DDPG is an off-policy algorithm, it uses a replay buffer to store past experiences for training. Training data is sampled randomly from the buffer to update the parameters.

Why are we able to train off-policy and use past experience which is obtained using outdated policies?? Because the Bellman equation does not care which transition tuples are used since the optimal value should satisfy the Bellman equation for all possible transitions.

Policy Learning Part of DDPG

In DDPG, the Q-value is used to learn the policy. The goal is to learn a deterministic policy which maximizes $Q_\phi (s,a)$. Since the action space is continuous, we perform gradient ascent with respect to policy parameters $\theta$:
$\underset{\theta}{max} \underset{s \sim D}{E} [Q_\phi(s,\mu_\theta (s))]$

The above image shows the training structure for the policy in DDPG.

Exploration Tricks

• To improve exploration throughout training, we can add noise to the policy
• Another trick to improve exploration at the beginning of training is to take actions which are sampled from a Continuous Uniform Distribution over valid actions, until a fixed number of steps is reached (start_steps). After that, we return to normal DDPG exploration with noise.

Pseudocode

References

1. https://spinningup.openai.com/en/latest/algorithms/ddpg.html