Reward and Return

Topics: Reinforcement Learning

Reward

The reward in RL is a function that outputs a number that indicates to the agent how good/bad the action taken was. Reward is usually a function of current state $s_t$, action taken $a_t$ and next state $s_{t+1}$:
$r_t=R(s_t, a_t, s_{t+1})$
However, for simplification, reward is sometimes taken to be a function of current state only $r_t=R(s_t)$ or current state-action pair $r_t=R(s_t, a_t)$.

Return

Return is the cumulative reward obtained by the agent over a trajectory $\tau$ of states, actions and next states. We distinguish between 2 types of returns:

Finite-Horizon Undiscounted Return

$R(\tau) = \sum_{t=0}^T r_t$
which is the sum of rewards obtained over a finite number of steps.

Infinite-Horizon Discounted Return

$R(\tau) = \sum_{t=0}^\infty \gamma^t \cdot r_t$
which is the sum of all the rewards ever obtained by the agent, discounted by a factor $\gamma \in [0, 1)$ depending on how far they are in the future.

Why do we need a discount factor?

1. Intuitively: cash now is better than cash later
2. Mathematically: the discount factor makes the sum finite instead of infinite, which makes it easier to deal with in equations

References

• https://spinningup.openai.com/en/latest/spinningup/rl_intro.html