Categorical Policy

Topics: Reinforcement Learning

Definition

A Categorical Policy is a type of Stochastic Policies that is used for discrete action spaces. It is like a classifier over discrete actions. Given the state as input, the neural network outputs the probability of selecting each action.
The neural network structure is as follows: the input is the state (or observation), followed by some convolutional or densely-connected layers, followed by a final layer that gives the logits for each action, and finally a softmax distribution to convert logits into probabilities.

Sampling

Given the probabilities for each action, sampling is nothing more than using a categorical distribution/multinomial distribution probability mass function.
This can be done with PyTorch or TensorFlow.

Log-Likelihood

The last layer which contains the probabilities for the actions can be thought of as a vector $P_\theta(s)$.
The log-likelihood for an action can be obtained by indexing the vector:
$log\;\pi_\theta(a|s)=log[P_\theta(s)]_a$

References

1. https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#