Bellman Equations

Topics: Reinforcement Learning

The Bellman equations are central for RL algorithms. The basic idea behind the Bellman equations is:
The value of your starting point is the reward you expect to get from being there, plus the value of wherever you land next.

The Bellman equations for the on-policy value functions are:

$$V^\pi (s) = \underset{\substack{a \sim \pi \\ \\ \\ s' \sim P(\cdot \vert s,a)}}{E} \left[ r(s,a) + \gamma \cdot V^\pi(s') \right]$$ $$Q^\pi (s,a) = \underset{s' \sim P(\cdot \vert s,a)}{E} \left[ r(s,a) + \gamma \cdot \underset{a' \sim \pi}{E} \left[ Q^\pi(s', a') \right] \right]$$

The Bellman equations for the optimal value functions are:

$$V^* (s) = \underset{a}{max} \; \underset{s' \sim P(\cdot \vert s,a)}{E} \left[ r(s,a) + \gamma \cdot V^*(s') \right]$$ $$Q^* (s,a) = \underset{s' \sim P(\cdot \vert s,a)}{E} \left[ r(s,a) + \gamma \cdot \underset{a'}{max} \; Q^*(s', a') \right]$$

References

• https://spinningup.openai.com/en/latest/spinningup/rl_intro.html