On Learning Symmetric Locomotion

Overview

The paper compares 4 approaches that incorporate symmetry into reinforcement learning specifically targeted to locomotion tasks.

A symmetric policy is defined by
$\pi_{\theta}(M_s(s)) = M_a(\pi_{\theta}(s))$
The value function should be invariant
$V_{\phi}(M_s(s))=V_{\phi}(s)$
A key distinction that the paper makes is that a symmetric policy does not guarantee a symmetric trajectory. If the character starts with its left foot forward, a symmetric policy may continue a gait in which the left foot remains the leading foot. The reflected initial condition would produce the corresponding right-leading trajectory. Thus, the policy can be perfectly symmetric even though each individual trajectory is asymmetric.

Methods

Method 1: Duplicate Tuples (DUP)

This is the most intuitive method where each trajectory is duplicated, mirrored, then added as an experience tuple along with the original. One drawback of this method is that the mirrored trajectories are not necessarily on-policy, which may affect the performance of on-policy methods like PPO and TRPO.

Method 2: Auxiliary Loss (LOSS)

This method creates a symmetry loss defined as
$\mathcal{L}_{sym}(\theta) = \sum_{t=1}^T || \pi_{\theta}(s_t) - M_a(\pi_{\theta}(M_s(s_t)))||^2$
and adds it as an auxiliary loss to the default PPO loss
$\pi_{\theta}= \underset{\theta}{\text{arg min}} \; \mathcal{L}_{PPO}(\theta) + w \mathcal{L}_{sym}(\theta)$
where $w$ is a hyperparameter that balances the two losses.

Method 3: Phase-Based Mirroring (PHASE)

The walking gait is divided into repeated gait cycles that are parametrized using a phase gait variable $\phi \in [0,1)$, where $\phi=0$ is the start of the gait and $\phi=1$ is the end of the gate, and $\phi$ wraps from 1 to 0.
To enforce symmetry, this method consists of learning the policy for the first half-cycle, and then replace by the mirrored states and actions during the second half-cycle:

$$a_t = \begin{cases} \pi_{\theta}(s_t) \quad \quad \quad \quad \quad \quad \quad 0 \leq \phi(s_t) < 0.5\\ M_a(\pi_{\theta}(M_s(s_t))) \;\,\, \quad \quad 0.5 \leq \phi(s_t) < 1 \end{cases}$$

Method 4: Symmetric Network Architecture (NET)

This method consists of enforcing symmetry at the network architecture. The paper proposes the approach in the following image. Details not included for brevity:

Results

A high level summary of results:

• all symmetry enforcement methods improve motion quality over the baseline
• the methods cannot be reliably ranked across different environments
• in general, DUP is the least effective
• the methods have no consistent and predictable impact on learning speed
• symmetric gaits are better achieved when any of the methods is applied

So, the authors suggest that symmetry methods be used for producing higher quality symmetric motions but not necessarily for faster learning.