Primacy Bias in DRL

Topics: Reinforcement Learning

Introduction

Primacy Bias refers to the tendency of Deep Reinforcement Learning algorithms to overfit to early interactions with the environment, which prevents the agent to effectively learn from subsequent data.

The idea is similar to human learning: the first experiences can have a long-lasting effect and can hinder learning new behaviors. For example, if a person learns a wrong technique when playing the guitar, the mind will unconsciously use it even if a new technique is being learned.

In RL, the effects of primacy bias are compounding because the agent collects its own training data by interacting with the environment. An agent that overfits to a poor policy will collect data of poor quality, which will make it more challenging to overcome the initial poor policy.

Motivating Experiments

Two experiments demonstrate the effect of primacy bias and motivate the solution:

1. 100 experiences are collected into a reply buffer and an SAC agent is trained $10^5$ times on this same data. This is an extreme case of overfitting to initial data. After that, the training of the agent continues normally, but it completely fails to learn anything. This proves the existence of primacy bias, where the agent was not able to recover from initial overfitting. In short, heavy priming causes unrecoverable overfitting.
   
2. An SAC agent is trained with a replay ratio of 9, i.e., 9 gradient updates per environment step, and fails. The replay buffer of that agent is then used to train a new agent from scratch, whcih performs really well. This shows that the issue is not with the data itself, but rather with the optimization that the agent performs on that data.
   

Proposed Solution

The previous examples lead naturally to the following solution: reset the last layers of the networks periodically while keeping the replay buffer intact. This way, the agent does not overfit to the previous learnings, and is then free to learn a better policy from the already collected data.
This solution introduces 2 design parameters:

• the number of layers to reset in the networks
• the frequency of the resets

For the SAC case, resetting the entire network was found to work best, and the resets were performed every $2 \times 10^5$ steps.

References

1. E. Nikishin, M. Schwarzer, P. D’Oro, P.-L. Bacon, and A. Courville, “The Primacy Bias in Deep Reinforcement Learning,” in Proceedings of the 39th International Conference on Machine Learning, PMLR, Jun. 2022, pp. 16828–16847. Accessed: May 21, 2026. [Online]. Available: https://proceedings.mlr.press/v162/nikishin22a.html