Jump Start RL

Topics: Reinforcement Learning

Motivation

A challenging problem in reinforcement learning is training a policy from scratch in environments with sparse rewards that require extensive exploration to reach the target goal. This is because a substantial amount of actions could be tried without receiving any rewards to guide the agent. To overcome this, offline-to-online approaches have been developed, in which an agent is first trained on an offline dataset, and is then fine-tuned via online interaction with the environment. This approach requires the availability of a full dataset to be able to train the value functions and the policy of the agent offline. If only a policy is available, the approach fails because the value network cannot be effectively trained.
Jump start RL is a general framework to train value-based RL algorithms using only a prior policy of any form.

JSRL Method

JSRL assumes access to a fixed prior policy, denoted guide policy $\pi^g(a|s)$. There are no assumptions on the guide policy, meaning that it could be from offline RL training, imitation learning, random search, or even a manually scripted. The RL policy being learned is called exploration policy $\pi^e(a|s)$.
The main idea is straightforward: in every horizon $H$, the guide policy is followed for the first $h$ steps, then the exploration policy is followed for the remainder of the episode. During the initial phases of training, the value of $h$ is close to $H$, meaning that the guide policy is followed for the most part of the episode. As training progresses and the exploration policy improves, $h$ is reduced gradually until it reaches $0$ and $\pi^e(a|s)$ trains over the full episode. This scheduling of the policies allows the agent to visit good and important states through the guide policy, and explore around them through the exploration policy.
The figure below summarizes the approach. In stage 1, the guide policy is followed almost the entire episode, and the agent explores near the goal state. This allows the agent to receive rewards while exploring because it is very close to the goal state. As training progresses, the use of the guide policy is reduced and replaced by the exploration policy, until it runs the full episodes. In contrast, in Vanilla RL, the agent must explore from the start and will not get any rewards until it reaches the goal state.

Two approaches are suggested to select the value of $h$ throughout training:

1. JSRL-Curriculum: the value of $h$ follows a fixed curriculum from $H$ to $0$. For example, these could be pre-defined values $H_1, H_2, ..., H_n$ or a decreasing function of training steps. In the paper, the value of $h$ changes to a new value every time the policy reaches a satisfactory performance measure $\beta$.
2. JSRL-Random: the value of $h$ is sampled uniformly from the set $\{H_1, H_2, ..., H_n\}$

Pseudocode

References

1. I. Uchendu et al., “Jump-Start Reinforcement Learning,” in Proceedings of the 40th International Conference on Machine Learning, PMLR, Jul. 2023, pp. 34556–34583. Accessed: Jun. 08, 2026. [Online]. Available: https://proceedings.mlr.press/v202/uchendu23a.html