Diagnosing Bottlenecks in Deep Q-learning Algorithms

Overview

The paper studies which commonly cited problems actually limit deep Q-learning. It uses a unit-testing framework that begins with exact Q-iteration and progressively introduces practical approximations, allowing the effects of function approximation, sampling error, non-stationarity, and sampling distributions to be examined separately.

Main Findings

• Function approximation: Divergence is uncommon when the function approximator is sufficiently expressive. Small networks instead introduce substantial bias, which bootstrapping can amplify.
• Sampling error: Q-learning is prone to overfitting. Replay buffers, larger and more diverse sample sets, and well-chosen early stopping can reduce it.
• Non-stationarity: Moving targets and distribution shift do not appear to be the primary causes of instability when isolated; approximation and sampling errors have a larger effect on performance.
• Sampling distributions: The choice of sampling or weighting distribution strongly affects solution quality. Broad, high-entropy distributions generally perform better than narrowly on-policy distributions, motivating adversarial feature matching as a practical weighting method.