Entropy-Regularized Reinforcement Learning

Topics: Reinforcement Learning

Entropy in General

Entropy is a measure of the randomness or uncertainty of a probability distribution.
Let $x$ be a random variable with probability mass or probability density function $p$. The entropy $H$ of $x$ is defined as (1):
$H(p)=\underset{x \sim p}{E} \Bigl[ -log \; p(x) \Bigr]$
So, for a discrete random variable $x$, the entropy is:
$H(p)= - \sum_{x} \; p(x) \; log \; p(x)$
And for a continuous random variable $x$, the entropy is:
$H(p)= - \int_{x} \; p(x) \; log \; p(x) \; dx$

Entropy in RL

In the case of the RL problem, the policy $\pi(a \vert s)$ defines the probability of taking action $a$ in state $s$; so, the policy is a probability distribution. Thus, entropy in RL is a measure of the randomness of the policy: the more random a policy is, the higher its entropy. A high entropy policy means that the agent takes varied actions, and thus, explores more. On the other hand, a low entropy policy means that the agent takes more predictable actions, which corresponds to lower exploration and higher exploitation.

The equation for the entropy of the policy is written as follows:
$H(\pi(\cdot \vert s)) = - \sum_{a} \pi(a \vert s) \; log \, \Bigl( \pi(a \vert s) \Bigr)$
The figure below compares a low entropy policy (left) to a high entropy policy (right) in a discrete action space (2).

The low entropy policy is almost deterministic, while the high entropy policy is highly stochastic.

Entropy-Regularized RL

In the Entropy-Regularized RL environment, entropy is used as an entropy bonus that gets added to the reward function. This is done for every time step, meaning that present actions also play a role in maximizing future entropy (2). The finite horizon discounted return in entropy-regularized RL is:
$R(\tau) = \sum_{t=0}^{\infty} \gamma^{t} \biggr( R(s_{t}, a_{t}, s_{t+1}) + \alpha H \Bigr( \pi(\cdot \vert s_{t}) \Bigr) \biggr)$
The entropy bonus can be seen as a regular reward term which encourages exploration. It can be controlled by $\alpha$, known as the entropy regularization coefficient.

If the entropy of the policy is large, and therefore the entropy bonus is large, while the original reward term is low, this means that little is known about the impact of different actions and more exploration is needed.

Also, in stochastic policy, action probabilities are proportional to their expected rewards. So, if entropy is very large relative to the rewards, then action probabilities are more or less equal, whereas if the entropy is small relative to the rewards, then the rewards are received from more defining action probabilities.

The optimal policy of the Entropy-Regularized RL problem becomes (1):
$\pi^{*} = arg\; \underset{\pi}{max} \underset{\tau \sim \pi}{E} \Biggl[ \sum_{t=0}^\infty \gamma^{t} \biggl( R(s_t, a_t, s_{t+1}) + \alpha H \Bigl( \pi( \cdot \vert s_t) \Bigr) \biggr) \Biggr]$

The value function is also changed to include the entropy bonuses from every timestep:
V^{\pi}(s) = \underset{\tau \sim \pi}{E} \Biggl[ \sum_{t=0}^\infty \gamma^{t} \biggl( R(s_t, a_t, s_{t+1}) + \alpha H \Bigl( \pi( \cdot \vert s_t) \Bigr) \biggr) \Bigg{\vert} s_{0} = s \Biggr]

The action value function is changed to include the entropy bonuses from every timestep except the first:
Q^{\pi}(s,a) = \underset{\tau \sim \pi}{E} \Bigg[ \sum_{t=0}^\infty \gamma^{t} \; R(s_t, a_t, s_{t+1}) + \alpha \sum_{t=1}^{\infty} \gamma^{t} H \Big( \pi( \cdot \vert s_t) \Big) \Bigg{\vert} s_{0} = s, \; a_0=a \Bigg]

The definitions of $V^{\pi}$ and $Q^{\pi}$ are connected by:
$V^{\pi}(s) = \underset{a \sim \pi}{E} \Big[ Q^{\pi}(s,a) \Big] + \alpha \, H \Big( \pi( \cdot \vert s_t) \Big)$

And the Bellman Equation for $Q^{\pi}$ is:

\begin{align*} & Q^{\pi}(s,a) = \underset{\substack{s' \sim P \\ \\ a' \sim \pi}}{E} \Bigg[ R(s, a, s') + \gamma \, \bigg( Q^{\pi}(s',a') + \alpha \, H \Big( \pi( \cdot \vert s_t) \Big) \bigg) \Bigg] \\ \\ & \quad\quad\quad\;\;\; = \underset{s' \sim P}{E} \bigg[ R(s,a,s') + \gamma \, V^{\pi}(s') \bigg] \end{align*}$$ ## Note The way we’ve set up the value functions in the entropy-regularized setting is a little bit arbitrary, and actually we could have done it differently (for example include the first entropy bonus in $Q^{\pi}$. The choice of definition may vary slightly across papers on the subject. ## References 1. https://spinningup.openai.com/en/latest/algorithms/sac.html#entropy-regularized-reinforcement-learning 2. https://towardsdatascience.com/entropy-regularized-reinforcement-learning-explained-2ba959c92aad/