Value Functions in RL

Topics: Reinforcement Learning

Definition

"Value" means the expected return when starting from a state or state-action pair. In other words, it indicates how good/bad it is to be in a current state or state-action.

The 4 Value Functions

There are 4 main "value" functions in RL:

The On-Policy Value Function

$V^\pi(s) = \underset{\tau \sim \pi}{E}[ R(\tau) \vert s_0=s]$
which gives the expected return if you start in a state $s$ and act according to the policy $\pi$.

The On-Policy Action-Value Function

$Q^\pi (s,a) = \underset{\tau \sim \pi}{E} [R(\tau) \vert s_0=s, a_0=a]$
which gives the expected return if you start in a state $s$, take an arbitrary action $a$ (which may not have come from the policy), and then forever follow policy $\pi$.

The Optimal Value Function

$V^*(s) = \underset{\pi}{max} \;\underset{\tau \sim \pi}{E}[ R(\tau) \vert s_0=s]$
which gives the expected return if you start in a state $s$ and always act according to the optimal policy $\pi$.

The Optimal Action-Value Function

$Q^* (s,a) = \underset{\pi}{max} \; \underset{\tau \sim \pi}{E} [R(\tau) \vert s_0=s, a_0=a]$
which gives the expected return if you start in a state $s$, take an arbitrary action $a$, and then forever follow the optimal policy.

Relations Between Value Functions

1. $V^\pi (s) = \underset{a \sim \pi}{E} [Q^\pi (s,a)]$
2. $V^* (s) = \underset{a}{max} \; Q^* (s,a)$
3. $a^* (s) = \underset{a}{arg\;max} \; Q^*(s,a)$

References

• https://spinningup.openai.com/en/latest/spinningup/rl_intro.html