BCQ

Topics: Reinforcement Learning

Problem Overview

Batch Constrained Q-Learning (BCQ) is an offline reinforcement learning algorithm. It is designed for the setting where an agent must learn from a fixed dataset of transitions $B = {(s, a, r, s')}$.
The agent is not allowed to collect new experience. This makes ordinary off-policy deep RL unstable, because the policy may choose actions that are not represented in the dataset.

Core Problem: Extrapolation Error

Standard Q-learning uses targets of the form:

$$y = r + \gamma \max_{a'} Q(s', a')$$

In continuous-action actor-critic methods such as DDPG, the target is often:

$$y = r + \gamma Q(s', \pi(s'))$$

The problem is that $a'$ or $\pi(s')$ may be an action that never appeared in the dataset. Since the Q-network has little or no evidence for that state-action pair, its value estimate can be arbitrary. The maximization step then tends to select overestimated, unsupported actions.

BCQ addresses this by constraining the policy to choose actions that are likely under the batch data.

Batch Constraint

In the ideal discrete case, a batch-constrained policy only chooses actions that appear in the dataset:

$$\pi(s) \in \{a : (s, a) \in B\}$$

The corresponding greedy improvement step becomes:

$$\pi'(s) = \arg\max_{a : (s, a) \in B} Q^\pi(s, a)$$

So the agent still improves with respect to Q-values, but it only compares actions supported by the data.

BCQ Approach

In continuous action spaces, the exact same $(s, a)$ pair is unlikely to appear many times. BCQ therefore learns an approximate action distribution from the dataset:

$$P_B(a \mid s)$$

Rather than maximizing over all possible actions, BCQ samples plausible actions from a learned generative model:

$$a_i \sim G_\omega(s)$$

where $G_\omega$ is a conditional variational autoencoder (CVAE) trained on dataset actions.

VAE Action Model

BCQ trains a conditional VAE to reconstruct dataset actions given states. For a transition action $a$ at state $s$, the encoder produces latent parameters:

$$\mu, \sigma = E_{\omega_1}(s, a)$$

Then a latent vector is sampled:

$$z \sim \mathcal{N}(\mu, \sigma)$$

and the decoder reconstructs the action:

$$\tilde{a} = D_{\omega_2}(s, z)$$

The VAE objective is:

$$\min_\omega \sum_{(s,a)\in B} \left[ \left(D_{\omega_2}(s, z) - a\right)^2 + D_{KL}\left(\mathcal{N}(\mu, \sigma) \,\|\, \mathcal{N}(0, I)\right) \right]$$

This teaches the model to generate actions similar to those in the dataset.

Perturbation Model

BCQ does not merely imitate the generated actions. It allows a small learned adjustment:

$$a_i' = a_i + \xi_\phi(s, a_i, \Phi)$$

where $\xi_\phi$ is a perturbation network and $\Phi$ bounds the maximum adjustment. This lets BCQ improve over the dataset while staying close to it.

The perturbation model is trained to increase the Q-value:

$$\max_\phi \sum_{s \in B} Q_\theta(s, a + \xi_\phi(s, a, \Phi)), \quad a \sim G_\omega(s)$$

BCQ Policy

At action-selection time, BCQ:

1. Samples $n$ candidate actions from the VAE.
2. Perturbs each candidate slightly.
3. Chooses the candidate with the highest Q-value.

The policy is:

$$\pi(s) = \arg\max_{a_i} Q_\theta(s, a_i + \xi_\phi(s, a_i, \Phi)), \quad \{a_i \sim G_\omega(s)\}_{i=1}^n$$

Critic and Value Updates

BCQ learns a critic $Q_\theta(s, a)$ using a bootstrapped target:

$$y = r + \gamma V_{\psi'}(s')$$

and minimizes:

$$\min_\theta \sum_{(s,a,r,s')\in B} \left( r + \gamma V_{\psi'}(s') - Q_\theta(s, a) \right)^2$$

The value network estimates the best supported action value:

$$V_\psi(s) \approx \max_i Q_\theta(s, a_i + \xi_\phi(s, a_i, \Phi)), \quad a_i \sim G_\omega(s)$$

It is trained with:

$$\min_\psi \sum_{s \in B} \left( \max_i Q_\theta(s, a_i + \xi_\phi(s, a_i, \Phi)) - V_\psi(s) \right)^2$$

Imitation-to-RL Tradeoff

BCQ has a useful continuum:

$$\Phi = 0,\ n = 1$$

resembles behavioral cloning, because the agent mostly follows generated dataset actions.
As:

$$\Phi \to a_{\max} - a_{\min}, \quad n \to \infty$$

the method approaches ordinary Q-learning, because the policy can search more of the action space.

The practical power of BCQ comes from choosing a middle ground: improve beyond the dataset when Q-values justify it, but avoid actions too far outside the data distribution.

Summary

BCQ solves a central offline RL problem: Q-functions are unreliable outside the dataset, and unconstrained maximization exploits those errors.
Its solution is to do Q-learning, but only over actions that look plausible under the batch data.

This makes BCQ more stable than standard off-policy deep RL in fixed-dataset settings, especially continuous-control tasks where unsupported actions are easy to generate and hard to evaluate correctly.

References

1. S. Fujimoto, D. Meger, and D. Precup, “Off-Policy Deep Reinforcement Learning without Exploration,” Aug. 10, 2019, arXiv: arXiv:1812.02900. doi: 10.48550/arXiv.1812.02900.