TD3

Topics: Reinforcement Learning

Introduction

A common failure mode of DDPG is the problem of Q-value overestimation. When it happens, it leads the policy to exploit it, which in turn breaks the policy.

Twin Delayed DDPG (TD3) is an algorithm based on DDPG that addresses this issue by employing the 3 tricks discussed below.

Similarly to DDPG, TD3 is a model-free, off-policy, actor-critic algorithm, that trains a deterministic policy aimed at environments with continuous action spaces.

Trick One: Target Policy Smoothing

In TD3, clipped noise is added to the actions that are sampled from the target policy when calculating the Q-learning targets. After adding the clipped noise, the target actions are clipped in order to remain in the valid action range. Therefore, the equation for the target action in TD3 is:
$a'(s')=clip \left[ \mu_{\theta_{tartget}}(s') + clip(\epsilon, -c, +c), a_{Low}, a_{High} \right]$
where $\epsilon \sim N(0,\sigma)$

Target policy smoothing makes it harder for the algorithm to exploit peaks in $Q$ because it smooths out $Q$ along changes in action.

Trick Two: Clipped Double Q-Learning

Compared to DDPG, TD3 learns two Q-functions $Q_{\phi_1}$and $Q_{\phi_2}$ instead of one, hence the name "twin". Both Q-functions use the same target value, which is calculated using whichever of the two Q-functions gives a smaller target value. Thus the equation of the target in TD3 is:
$y(r,s',d) = r + \gamma \cdot (1-d) \cdot \underset{i=1,2}{min} \; Q_{\phi_{target}}(s', a'(s'))$
As stated earlier, both networks are updated with that same target:
$L(\phi_1, D) = \underset{(s,a,r,s',d) \sim D}{E} \left[ Q_{\phi_1}(s,a) - y(r,s',d) \right] ^2$
$L(\phi_2, D) = \underset{(s,a,r,s',d) \sim D}{E} \left[ Q_{\phi_2}(s,a) - y(r,s',d) \right] ^2$
Using the smaller Q-value in clipped double Q-learning reduces overestimation of the Q-function.

Trick Three: "Delayed" Policy Updates

In TD3, the policy and target networks are updated less frequently than the Q-function (critic) network. This reduces the volatility that arises in DDPG because of how a policy update changes the target. Also, the policy is updated by maximizing $Q_{\phi_1}$ only, just like DDPG:
$\underset{\theta}{max} \underset{s \sim D}{E} \left[ Q_{\phi_1}(s, \mu_{\theta}(s)) \right]$

Pseudocode

References

• https://spinningup.openai.com/en/latest/algorithms/td3.html