Dormant Neurons in DRL

Topics: Reinforcement Learning

Introduction

The Dormant Neuron Phenomenon in Deep Reinforcement Learning refers to the observed empirical phenomenon where an increasing number of an agent's neurons become inactive during training. In other words, the activation of neurons becomes so small such that they barely affect the output of the layer. This phenomenon is not observed in traditional supervised learning; on the contrary, most neurons remain active throughout training.

The dormant neuron phenomenon poses 2 questions:

• Is the training of RL agents neural networks fundamentally different from regular supervised learning, especially with the non-stationarity in RL data?
• Do RL agents use neural network parameters to their full potential?

Definition

Given an input distribution $D$, let $h_i^l(x)$ denote the activation of neuron $i$ in layer $l$ under input $x \in D$ and $H^l$ be the number of neurons in layer $l$. The score of a neuron is defined via the normalized average of its activation as follows:

$$s_i^l = \frac{\mathbb{E}_{x \in D} |h_i^l(x)|}{\frac{1}{H^l} \sum_{k \in h} \mathbb{E}_{x \in D} |h_k^l(x)|}$$

A neuron is said to be $\tau$-dormant if $s_i^l \leq \tau$.

An algorithm exhibits the dormant neuron phenomenon if the number of $\tau$-dormant neurons increases steadily over training.

Observations

• The target non-stationarity exacerbates the dormant neuron phenomenon
• Input non-stationarity does not appear to be a major factor
• Dormant neurons mostly remain dormant
• More gradient updates (higher replay ratio) lead to more dormant neurons

Proposed Solution

The proposed solution, called Recycling Dormant Neurons, consists of periodically checking for $\tau$-dormant neurons and reinitializing their weights.

References

1. G. Sokar, R. Agarwal, P. S. Castro, and U. Evci, “The Dormant Neuron Phenomenon in Deep Reinforcement Learning,” in Proceedings of the 40th International Conference on Machine Learning, PMLR, Jul. 2023, pp. 32145–32168. Accessed: May 21, 2026. [Online]. Available: https://proceedings.mlr.press/v202/sokar23a.html