Evaluation of RL Algorithms

Topics: Reinforcement Learning

Overview

This page discusses the process of evaluating Reinforcement Learning algorithms and the factors that affect the results and their reproducibility.
The main question to be answered when evaluating a RL algorithm is which algorithm(s) perform well across a wide range of environments with little or no environment-specific tuning?

Data Collection

The data collection phase consists of running full training and evaluation trials in the benchmarking environments and recording the episode returns.

A full data collection process is as follows:

• run a total of 15 trials, each with different training and evaluation seeds
• training runs for a preset number of timesteps, for example $10^6$ steps for easy/medium environment and $3\times10^6$ to $5\times10^6$ steps for more difficult environments
• run evaluation episodes at periodic intervals during training. For example, every $10\times10^3$ steps, 5 evaluation episodes are run and the mean and standard deviation of the episode returns is stored.
• the random seeds should be different between training and evaluation and across trials. For example, when running 15 trials with 8 parallel workers in each trial, the random seeds can look as follows:
  • Trial 0: worker seeds 0, 1, 2, … 7 and evaluation seed 1000
  • Trial 1: worker seeds 8, 9, 10, … 15 and evaluation seed 1001
  • Trial 2: worker seeds 16, 17, 18, … 3 and evaluation seed 1002
  • …
  • Trial 14: worker seeds 112, 113, 114, … 119 and evaluation seed 1014

The data collection procedure results in training and evaluation learning curves for every trial

Data Processing

The data processing phase consists of calculating some Performance Metrics in RL from the evaluation results, plotting the learning curves with confidence intervals, and analyzing the performance.

Factors Affecting Results and Reproducibility

The literature reports a wide range of results for the same baseline algorithms and environments. Also, reproducing the results of state-of-the-art papers is not a straightforward task. Below is a list of the factors that affect the results and reproducibility of RL experiments (1).

• Hyperparameters: tuning the hyperparameters plays a crucial role in extracting the best performance from many RL algorithms
• Network Architecture: for deep RL algorithms, performance can be significantly affected by simple changes to the policy or value networks architecture and activation functions.
• Reward Scale: reward rescaling consists of multiplying the reward by a constant factor before storing it in the replay buffer or using it for training. This can have a large effect, especially for environments where the reward is not scaled, like OpenAI Gym.
• Random Seeds: the random seed used to set up the trial has a significant effect on the training process and results. Therefore, 10-15 trials, each with a different random seed, are reported in the literature. Also, the reported results can be distorted by averaging a different subset of trials.
• Environments: some environments have more stable dynamics than others. For instance, cheetah is more stable than hopper. Therefore, the algorithms that perform well in one do not necessarily perform well in the other. So, it is crucial to test algorithms across a wide range of environments with different characteristics.
• Codebases: implementation differences of the same algorithm can have drastic impacts on performance
• Hardware: there are differences in results, although not very significant, from running on different hardware, for example CPU vs. GPU. Also, based on my own experiments, there are small differences in numbers from running on CPU on different machines.

Research on Quantitative Evaluation

Because hyperparameters are a significant factor in the performance of RL algorithms, reference (2) emphasizes the importance of eliminating the role of manual tuning of hyperparameters in the benchmarking process. To do so, the authors suggest to specify a range for each hyperparameter and to randomly sample them before every trial. The advantage of this approach is that it allows to compare the effect of hyperparameters on different algorithms and the easiness of using algorithms with minimal tuning. The downside is that the approach requires a high number of trials to generate statistically significant results; 10-15 trials are not enough.

Moreover, there is a need for a unified, quantitative, composite metric to rank the overall performance of algorithms across all the environments. Reference (2) suggests an approach that computes a score for each algorithm based on a weighted sum of projections on the CDF of results. The weight for each environment in the composite metric is calculated via a suggested game theoretic approach that seems complicated. The authors also suggest a way to compute the uncertainty of the calculated metric. The advantage is that the suggested method is a standard way to evaluate algorithms across a wide range of environments, while the downside is that a high number of trials is required to obtain meaningful CDFs.

References

1. P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep Reinforcement Learning that Matters”.
2. S. M. Jordan, Y. Chandak, D. Cohen, M. Zhang, and P. S. Thomas, “Evaluating the Performance of Reinforcement Learning Algorithms”.