Reinforcement-Learning-Enhanced Model Predictive Control with Application to Autonomous Planetary Landing

This paper introduces a hybrid RL-MPC approach called Reinforcement Learning Enhanced MPC (RLE-MPC) and demonstrates it on the guidance and control for pinpoint landing of a lunar lander.

The approach aims to benefit from the strengths of RL and MPC and address their respective downsides:

• RL handles uncertainty and long-term planning well
• RL does not guarantee the satisfaction of hard constraints
• MPC plans on shorter horizons
• MPC generally uses a nominal model and does not handle uncertainty
• MPC includes hard constraints explicitly in the problem formulation

The concept of the RLE-MPC method is straightforward. The RL policy learns and outputs the coefficients of the cost function of the optimization problem of MPC. This way, the RL agent modifies the cost function of the MPC problem to take into account learned uncertainty and long-term planning, and the control action is solved by MPC, which maintains the desired properties of MPC. This way, the MPC maintains the nominal dynamics equations, while RL trains on the MDP with uncertainties and embeds the information into the MPC cost function parameters.

The figure below shows the structure of RLE-MPC

RLE-MPC is a general framework and can be applied to lots of optimal control problems. In this paper, it is tested on a 6 DOF pinpoint lunar lander problem. Since the problem is non-convex, the paper presents all the model equations, the full derivations of lossless convexification, as well all the parameters used in the simulation. So, no need to copy them here.