Double Deep Q-Learning

Topics: Reinforcement Learning

Concept

Double Deep Q-Learning (Double DQN) is an improvement of the Deep Q-Learning algorithm. It aims to address the issue of Overestimation in Q-Learning.

The concept is straightforward. Double DQN uses two independent Q-value networks $Q_{\phi_1}$ and $Q_{\phi_2}$. The Q-network being trained is used to select the maximizing action (action having the highest Q-value), while the other network is used to evaluate the action (obtain its estimated Q-value) (1). In other words, to train $Q_{\phi_1}$, we select the action with the highest Q-value from $Q_{\phi_1}$ but use its Q-value from $Q_{\phi_2}$. So, the target to train $Q_{\phi_1}$ is:
$y = r + \gamma \cdot Q_{\phi_2} \left( s', \underset{a'}{arg\, max} \; Q_{\phi_1}(s', a') \right)$
Similarly, in order to train $Q_{\phi_2}$, we select the action with the highest Q-value from $Q_{\phi_2}$ but use its Q-value from $Q_{\phi_1}$. So, the target to train $Q_{\phi_2}$ is:
$y = r + \gamma \cdot Q_{\phi_1} \left( s', \underset{a'}{arg\, max} \; Q_{\phi_2}(s', a') \right)$

Note: this is different from the DQN with target network trick to avoid shifting targets. If we combine target networks and Double DQN, we end up with 4 networks: 2 main and 2 targets (similar to TD3) (1).

References

1. https://www.youtube.com/watch?v=Lb5ADHnRQV8&list=PLsStLNVyGngTPLwDCt8G1J9-iYNBNj53K&index=1&t=2s