The Reinforcement Learning Problem

Topics: Reinforcement Learning

Reinforcement Learning (RL) is the study of learning by trial and error to achieve a certain goal. It employs the concept of reward and penalty to reinforce or change the behavior of the agent.

The goal of the agent in RL is to select/learn a policy which maximizes the expected return when the agent acts according to it.

A few equations here to make it concrete. Consider a stochastic policy and stochastic environment transitions. The the probability of a trajectory is:
$P(\tau \vert \pi) = \rho_0(s_0) \cdot \prod_{t=0}^{T-1} P(s_{t+1} \vert s_t, a_t) \cdot \pi(a_t \vert s_t)$
where:
$\rho_0(s_0)$: probability of the initial state
$P(s_{t+1} \vert s_t, a_t)$: state transition probability, which is the probability of moving to a new state given the current state and the selected action
$\pi(a_t \vert s_t)$: stochastic policy, which represents the probability of selecting an action given the current state

The expected return is:

$$\begin{align*} & J(\pi) = \underset{\tau \sim \pi}{E}[R(\tau)] \\ \\ & J(\pi) = \int_\tau P(\tau \vert \pi) \cdot R(\tau) \end{align*}$$

where $R(\tau)$ is the return.

Then the central optimization problem in RL can be expressed as:
$\pi^* = \underset{\pi}{arg\;max}[J(\pi)]$
where $\pi^*$ is the optimal policy, which is the policy that maximizes the expected return $J(\pi)$.

References

• https://spinningup.openai.com/en/latest/spinningup/rl_intro.html