Phased Actor in Actor-Critic (PAAC)

Topics: Reinforcement Learning

Introduction

Phased Actor in Actor-Critic (PAAC) is a proposed mechanism that can be added to policy gradient actor-critic algorithms like DDPG, TD3, and D4PG. It aims to address the problem of high policy gradient variance and to encourage exploration and improve the overall learning performance.

In continuous control problems with deterministic policy, a common approach to estimate the policy gradient is to use the Q-value. This is the case in DDPG and TD3 for example. This method is straightforward but can introduce high variance in the policy gradient. Another approach is to use the Temporal Difference error (TD error). In comparison to Q-value, this method results in a lower variance; however, it is slightly more complex and can result in insufficient exploration.

PAAC introduces a switching mechanism for estimating the policy gradient and updating the actor network where, at the beginning of training, the agent uses the Q-value for updates, whereas, as training evolves, the agent switches to using TD error. This allows the agent to explore more at the beginning and to converge nicely with lower variance towards the end of the episode.

Mathematical Model

This section explains the switching mechanism, where Q-value plays a dominant role at the beginning of training and TD error dominates the end of training. The switching function is chosen to be a design hyperparameter in order to test and compare different functions.

Let $\theta$ be the parameters of the actor network and $B$ be the memory buffer. Then the PAAC policy gradient is defined as:
$g=\underset{R,x_{k} \sim B}{E} \Bigr[ \nabla_{\theta} \Psi^{k}(\theta) \Bigr]$
where

$$\Psi^{k}(\theta)= \begin{cases} Q \Bigr( x_{k}, \pi(x_{k} \vert \theta) \Bigr) && \text{if } \omega \leq M(k) \\ \\ \delta(\theta) && \text{if } \omega > M(k) \end{cases}$$

$M(k)$ is the switching function and is designed to be a monotonically decreasing function from 1 to 0. Three switching functions are studied:

• linear: $M(k)=1 - \frac{k}{K_{total}}$
• quadratic: $M(k)=(1- \frac{k}{K_{total}})^2$
• hard switch: which is a step function from 1 to 0 halfway through the episode

$\omega \in [0,1]$ is a uniformly distributed random variable.

The TD error $\delta(\theta)$ is the same that is used in updating the critic network:
$\delta(\theta) = Q \Bigr( x_{k}, \pi(x_{k} \vert \theta) \Bigr) - y$
$y=\gamma \, Q' \Bigr( x_{k+1}, \pi'(x_{k+1}) \Bigr) + R(x_{k}, u_{k})$

Pseudocode

Results

• PAAC improves the performance of base algorithms most of the time
• The effects of PAAC seem more pronounced in hard/complex tasks compared to easy tasks
• The effect of different switching functions depends on the state of the learning process. If we switch from Q-value to TD too early, especially for complex tasks, then the agent would not have explored enough, and the performance is not as good. And if we delay the switch from Q-value to TD too much, then we are not fully taking advantage of the power of PAAC in reducing variance.

References

1. R. Wu, J. Zhong, and J. Si, “Actor-Critic Reinforcement Learning with Phased Actor,” arXiv.org, 2024. https://arxiv.org/abs/2404.11834