Performance Metrics in RL

Topics: Reinforcement Learning

Experiment Description

An episode is defined as one full run of the training environment. It starts at the initial state and keeps running until the agent reaches a terminal state (car crash for example) or the limit for number of steps is reached (1000 steps for example). During an episode, the stage rewards/costs over all the time steps are added to obtain the total reward.

A learning trial consists of running $d$ consecutive episodes one after the other while training the same policy.

A full experiment consists of $n$ learning trials, each starting from a different random seed.

In every learning trial, evaluations are performed every $k$ episodes. Every evaluation consists for running $x$ evaluation episodes without exploration noise and averaging their total rewards.

An example of a full training experiment is as follows:
Use the acrobot swingup environment from DeepMind Control Suite. Every episode consists of a 1000 steps, or less if a terminal state is reached earlier. 10 trials are run, each starting with a different random seed. Every trial consists of $10^6$ steps, and 5 evaluation episode are run after every $10\times10^3$ training steps.

Performance Metrics

• Average reward curve: the total reward for the evaluation episodes averaged to get a learning curve of the agent: average evaluation reward vs. number of training episodes.

• Total cost: this can be seen as the reward/cost for the final version of the agent. It is calculated as the average total cost from the last $y$ evaluations over all the trials

• Variance: this is the variance of the values that are averaged to obtain the total cost above

• Convergence speed: using area under the curve. If the problem is to maximize rewards, then the bigger the area, the faster the agent learned. If the problem is to minimize costs, then the smaller the area under the curve, the faster the agent learned.

• Success rate: when the agent reaches a specific reward threshold, the trial is considered as a success

• Coefficient of variation: $cv(Q) = \frac{std(Q)}{Q} \%$

• Overestimation bias: compare Q-value estimates to actual returns from the environment (Monte Carlo) to check for overestimation bias

References

1. Zhong, Junmin, Ruofan Wu, and Jennie Si. "A Long N-step Surrogate Stage Reward for Deep Reinforcement Learning." Advances in Neural Information Processing Systems 36 (2023): 12733-12745.
2. R. Wu, J. Zhong, and J. Si, “Actor-Critic Reinforcement Learning with Phased Actor,” arXiv.org, 2024. https://arxiv.org/abs/2404.11834