Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates

Overview

This is a diagnostics paper that aims to study the question: when and why does Data Augmentation (DA) improve data efficiency in reinforcement learning? More specifically, they focus on off-policy RL, and on environments with sparse rewards where they can perform multiple augmentation techniques (like translation and rotation) on the same transition.

Off-Policy DA Framework

They introduce a general off-policy DA framework to conduct experiments that they can analyze. They use 2 replay buffers:

• the observed replay buffer, which stores the transitions that are obtained from the interaction of the agent with the environment
• the augmented replay buffer, which stores the augmented transitions generated from the observed ones

Then, to control the amount of augmented data that is generated and used in each update, they introduce some parameters:

• the augmentation ratio $m$, which specifies the number of augmented transitions generated per observed transition
• the update ratio $\alpha$, which denotes the ratio of augmented to observed data used in each mini-batch for updates
• the augmented replay ratio $\beta$, which denotes the number of updates per augmented transition generated

Disentangling Properties of DA

The paper identifies 3 aspects of data augmentation that may impact its effectiveness, and aim to understand the effect of each on the performance of DA:

• State-Action Coverage: augmentation created valid transitions in parts of the state-action space that the policy has not visited much
• Reward Density: augmentation creates more transitions containing reward signal, especially useful in sparse-reward tasks
• Augmented Replay Ratio: it may be beneficial to lower the augmented replay ratio by generating as many augmented transitions as possible, meaning that the agent gets more diverse augmented data per update

Findings

• Data augmentation improves sample efficiency over collecting the same amount of experiences with the agent's policy
• Increasing state-action coverage via DA often has a much greater impact on data efficiency than increasing reward density; it is the primary benefit in most tasks. However, increased reward density can also play a role
• Decreasing the augmented replay ratio alone substantially improves data efficiency and overall performance. In other words, using many diverse augmented samples, so that each one is replayed less often, can dramatically improve learning.

Limitations

• focused on low-dimensional state-based and sparse-reward tasks with continuous actions; so, findings may differ for tasks with dense rewards, discrete actions, and visual observations
• focused on model-free RL only