Expected SARSA

Topics: Reinforcement Learning

Algorithm Overview

Expected SARSA is a tabular, on-policy reinforcement learning algorithm that learns a discrete policy. It is targeted at environments with discrete state and discrete action spaces.

It can be thought as being like Q-Learning but with expected value instead of maximum over next state-action pair.

The update rule for Expected SARSA is as follows (1):

$$\begin{align*} & Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha \, \biggr[ r_{t+1} + \gamma \, \underset{\pi}{\mathbb{E}} \Bigr[ Q(s_{t+1}, a_{t+1}) \Bigr\vert s_{t+1} \Bigr] - Q(s_t, a_t) \biggr] \\ \\ & Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha \, \biggr[ r_{t+1} + \gamma \, \Bigr[ \sum_{a} \pi(a \vert s_{t+1}) \, Q(s_{t+1}, a) \Bigr] - Q(s_t, a_t) \biggr] \end{align*}$$

Pseudocode

(2)

References

1. R. S. Sutton and A. G. Barto, Reinforcement learning: an introduction. in Adaptive computation and machine learning. Cambridge, Mass: MIT Press, 2018.
2. https://jochemsoons.medium.com/a-comparison-between-sarsa-and-expected-sarsa-66b931202c75