Leveraging Symmetry to Accelerate Learning of Trajectory Tracking Controllers for Free-Flying Robotic Systems

Overview

The paper uses known Lie group symmetries of free-flying robots to reduce the state space in which a trajectory tracking policy is learned. A standard RL policy observes the actual state, reference state, and reference action independently, even though many such observations describe equivalent tracking problems. For example, translating both the robot and its reference does not change their relative configuration or the required feedback correction.

The authors show that symmetry in the dynamics and running costs induces an MDP homomorphism from the tracking MDP to a lower-dimensional quotient MDP. A policy can be trained with standard RL on this quotient and lifted back to the original system without loss of optimality. The reduced observations therefore encode only task-relevant relative quantities rather than requiring the policy to learn invariance from data.

Tracking Control Problem

A tracking control problem is defined as

$$\mathcal{T}=(\mathcal{X},\mathcal{U},f,J_{\mathcal{X}},J_{\mathcal{U}},\rho,\gamma),$$

where $\mathcal{X}$ and $\mathcal{U}$ are the physical state and action spaces, $f(\cdot\mid x,u)$ is the system dynamics, $J_{\mathcal{X}}(x,x^{\mathrm d})$ is the tracking cost, $J_{\mathcal{U}}(u,u^{\mathrm d})$ is the control effort cost, $\rho$ is a distribution over reference actions, and $\gamma$ is the discount factor.

It induces the stationary tracking MDP

$$\mathcal{M}_{\mathcal{T}}=(\mathcal{S},\mathcal{A},R,\tau,\gamma), \qquad \mathcal{S}=\mathcal{X}\times\mathcal{X}\times\mathcal{U}, \quad \mathcal{A}=\mathcal{U}.$$

The state contains the actual state, reference state, and reference action:

$$s_t=(x_t,x_t^{\mathrm d},u_t^{\mathrm d}), \qquad a_t=u_t.$$

The reward penalizes tracking and action error:

$$R((x,x^{\mathrm d},u^{\mathrm d}),u) =-J_{\mathcal{X}}(x,x^{\mathrm d})-J_{\mathcal{U}}(u,u^{\mathrm d}).$$

The actual and reference systems evolve under the same dynamics, while the next reference action is sampled from $\rho$:

$$\begin{aligned} x_{t+1}&\sim f(\cdot\mid x_t,u_t),\\ x_{t+1}^{\mathrm d}&\sim f(\cdot\mid x_t^{\mathrm d},u_t^{\mathrm d}),\\ u_{t+1}^{\mathrm d}&\sim\rho. \end{aligned}$$

Including the reference in the state produces one time-invariant MDP over a family of reference trajectories. In contrast, defining the cost using one fixed reference $x_t^{\mathrm d}$ would produce a non-stationary MDP whose optimal policy is tied to that particular trajectory. At deployment, the random reference controls can be replaced by a planned sequence of dynamically feasible reference states and actions.

MDP Homomorphism Induced by Lie Group Symmetries

Consider an original MDP $\mathcal{M}$ and a quotient MDP $\widetilde{\mathcal{M}}$. A continuous MDP homomorphism consists of a state map $p$ and a state-dependent action map $h$ satisfying

$$R(s,a)=\widetilde{R}(p(s),h(s,a))$$

and

$$\tau(p^{-1}(\widetilde{B})\mid s,a) =\widetilde{\tau}(\widetilde{B}\mid p(s),h(s,a)).$$

Thus, the abstraction preserves rewards and transition probabilities between abstract states. It also preserves values:

$$Q^{\pi}(s,a)=\widetilde{Q}^{\widetilde{\pi}}(p(s),h(s,a)),$$

and a lift of an optimal quotient policy is optimal in the original MDP.

Let a Lie group $\mathcal{G}$ act on states and actions through $\Phi_g(s)$ and $\Psi_g(a)$. It is an MDP symmetry when symmetry-related state-action pairs have the same reward and the same transition probabilities between state orbits. The orbit

$$\Phi_{\mathcal{G}}(s)=\{\Phi_g(s):g\in\mathcal{G}\}$$

contains all states equivalent to $s$, and the quotient space $\mathcal{S}/\mathcal{G}$ contains one element per orbit.

If the group action on states $\Phi$ is free and proper and there is an equivariant map $\lambda:\mathcal{S}\rightarrow\mathcal{G}$, the symmetry induces the homomorphism

$$p(s)=\Phi_{\mathcal{G}}(s), \qquad h(s,a)=\Psi_{\lambda(s)^{-1}}(a).$$

Here, $p$ removes the symmetry coordinates and $h$ expresses the action in the corresponding normalized frame. The quotient MDP is

$$\widetilde{\mathcal{M}} =\left(\mathcal{S}/\mathcal{G},\mathcal{A},\widetilde{R},\widetilde{\tau},\gamma\right),$$

with

$$\widetilde{R}(\widetilde{s},\widetilde{a}) =R\left(s,\Psi_{\lambda(s)}(\widetilde{a})\right), \qquad s\in p^{-1}(\widetilde{s}),$$

and

$$\widetilde{\tau}(\widetilde{B}\mid\widetilde{s},\widetilde{a}) =\tau\left(p^{-1}(\widetilde{B})\mid s, \Psi_{\lambda(s)}(\widetilde{a})\right), \qquad s\in p^{-1}(\widetilde{s}).$$

Symmetry guarantees that these expressions do not depend on which representative $s$ of the orbit is chosen. For a deterministic quotient policy, the lifted policy is

$$\pi^{\uparrow}(s) =\Psi_{\lambda(s)}\left(\widetilde{\pi}(p(s))\right).$$

Particle Example

The particle has state $x=(r,v)\in T\mathbb{R}^{3}$, action $u\in\mathbb{R}^{3}$, and discrete dynamics

$$r_{t+1}=r_t+v_t\,\mathrm{dt}, \qquad v_{t+1}=v_t+\frac{1}{m}u_t\,\mathrm{dt}.$$

Its tracking MDP state is

$$s=(r,v,r^{\mathrm d},v^{\mathrm d},u^{\mathrm d}),$$

with reward

$$R(s,u) =-\alpha(r-r^{\mathrm d}) -c_v\lVert v-v^{\mathrm d}\rVert -c_u\lVert u-u^{\mathrm d}\rVert.$$

Here, $\alpha$ is the position-error penalty used by the paper.

The MDP is invariant to common shifts in actual and reference position and velocity, and to common shifts in actual and reference force. The symmetry group is

$$\mathcal{G}=T\mathbb{R}^{3}\times\mathbb{R}^{3},$$

acting as

$$\Phi_{(k,h)}(s) =(r+k_1,v+k_2,r^{\mathrm d}+k_1,v^{\mathrm d}+k_2,u^{\mathrm d}+h),$$ $$\Psi_{(k,h)}(u)=u+h.$$

Choosing

$$\lambda(s)=((r^{\mathrm d},v^{\mathrm d}),u^{\mathrm d})$$

gives the quotient state and action

$$\widetilde{s}=p(s)=(r-r^{\mathrm d},v-v^{\mathrm d}) =(r^{\mathrm e},v^{\mathrm e}),$$ $$\widetilde{u}=h(s,u)=u-u^{\mathrm d}.$$

The quotient reward is

$$\widetilde{R}(\widetilde{s},\widetilde{u}) =-\alpha(r^{\mathrm e}) -c_v\lVert v^{\mathrm e}\rVert -c_u\lVert\widetilde{u}\rVert,$$

and the quotient transitions are the error dynamics

$$r_{t+1}^{\mathrm e}=r_t^{\mathrm e}+v_t^{\mathrm e}\,\mathrm{dt}, \qquad v_{t+1}^{\mathrm e}=v_t^{\mathrm e}+\frac{1}{m}\widetilde{u}_t\,\mathrm{dt}.$$

A policy can therefore be learned using only position and velocity error, then lifted through

$$u_t=u_t^{\mathrm d} +\widetilde{\pi}(r_t-r_t^{\mathrm d},v_t-v_t^{\mathrm d}).$$

The resulting controller has the familiar structure of reference feedforward plus learned error feedback, derived here as an optimality-preserving MDP quotient.