Replay Buffer Parameters

Topics: Reinforcement Learning

Introduction

A replay buffer (or experience replay) is a fixed-size memory that stores transitions collected by an RL agent during interaction with the environment. Instead of learning only from the newest experience, the agent samples past transitions from this buffer for training. It is necessary for off-policy algorithms like DDPG, TD3 and SAC.

Experience replay improves:

• sample efficiency by reusing data multiple times,
• training stability by decorrelating updates,
• and enables off-policy learning.

The replay buffer is typically implemented as a circular buffer:

• new transitions are added continuously,
• old transitions are discarded once capacity is exceeded.

Parameters

Replay Capacity

Replay capacity is the total number of transitions that can be stored in the replay buffer at a given time. A larger replay capacity requires more memory.

Age of the Oldest Policy

The age of a transition is the number of gradient updates that have occurred since the transition was generated. The age of the oldest policy in the replay buffer is therefore the age of the oldest transition currently stored in memory.
The age of the oldest policy can be interpreted as the degree of "off-policyness" of the buffer. Intuitively, the older a policy is, the more likely it is to be different from the current policy. Equivalently, the lower the age of the oldest policy, the closer the learning process is to on-policy.

Replay Ratio

Replay ratio is the number of gradient updates performed per environment transition collected.

$\text{Replay Ratio} = \frac{\text{Gradient Updates}}{\text{Environment Transitions}}$

Relation Between Parameters

Replay capacity, replay ratio, and age of the oldest policy are tightly coupled. Changing one of them generally affects at least one of the others.

• If replay ratio is held constant, increasing replay capacity increases the age of the oldest policy. This happens because transitions remain in memory longer before being overwritten. Larger buffers therefore contain older data and become more off-policy. For example, if replay ratio is set to 1 and the buffer capacity is $10^6$, then the age of the oldest policy is $10^6$. If the replay ratio is kept constant, and the buffer capacity is increased to $2 \times 10^6$, then the age of the oldest policy increases to $2 \times 10^6$.

• If the age of the oldest policy is held constant, increasing replay capacity requires collecting more transitions for a given policy. Since more environment interaction is needed relative to the number of learning updates, the replay ratio decreases.

• If replay capacity is held constant, decreasing the age of the oldest policy requires replacing stored transitions more quickly. This also requires collecting more environment transitions relative to gradient updates, which again decreases replay ratio.

Replay ratio can therefore be interpreted as a measure of how aggressively the agent reuses past experience. Higher replay ratios correspond to more learning updates per collected transition, while lower replay ratios correspond to more frequent collection of fresh data.

These parameters create a tradeoff between data diversity, freshness of experience, and sample reuse. Larger replay capacities improve state-action coverage and diversity of experience, while smaller ages of the oldest policy keep learning closer to on-policy behavior. Replay ratio determines how strongly the learner emphasizes reusing existing data versus collecting new experience.

References

1. W. Fedus et al., “Revisiting Fundamentals of Experience Replay,” in Proceedings of the 37th International Conference on Machine Learning, PMLR, Nov. 2020, pp. 3061–3071. [Online]. Available: https://proceedings.mlr.press/v119/fedus20a.html