Policy

Topics: Reinforcement Learning

Action Space

The action space is the set of all the valid actions that can be taken. We differentiate between:

• discrete action space: finite set of actions
• continuous action space: infinite set of actions

Policy

The policy is the rule that is used by the agent to decide what action to take. We distinguish between:

• deterministic policy: $a_t=\mu(s_t)$
• stochastic policy: $a_t\sim\pi(.|s_t)$
  In deep RL, we deal with parametrized policies, which are functions that depend on a set of parameters, most commonly a neural network.
  $a_t=\mu_\theta(s_t)$
  $a_t\sim\pi_\theta(.|s_t)$

1. Deterministic Policy

The goal of the agent is to learn one deterministic action for every state. In deep RL, is it done using a neural network.

The state and action can be 1-dimensional or k-dimensional.

2. Stochastic Policy

The two most common types of stochastic policies are:

• Categorical Policy: for discrete action spaces
• Diagonal Gaussian Policy: for continuous action spaces

There are two crucial computations needed to use and train stochastic policies:

• sampling actions from the policy
• computing the log-likelihood of particular actions $log\;\pi_\theta(a|s)$

References

1. https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#