Allen's REINFORCE notes

Allen's REINFORCE notes

Links

• RLbook2020

Motivation

Recall that the objective of Reinforcement Learning is to find an optimal policy $\pi ^{*}$ which we encode in a neural network with parameters $\theta ^{*}$. These optimal parameters are defined as $\theta ^{*}={\text{argmax}}_{\theta }E_{\tau \sim p_{\theta }(\tau )}\left[\sum _{t}r(s_{t},a_{t})\right]$. Let's unpack what this means. To phrase it in english, this is basically saying that the optimal policy is one such that the expected value of the total reward over following a trajectory determined by the policy is the highest over all policies.

Learning

Learning involves the agent taking actions and the environment returning a new state and reward.

• Input: $s_{t}$: States at each time step
• Output: $a_{t}$: Actions at each time step
• Data: $(s_{1},a_{1},r_{1},...,s_{T},a_{T},r_{T})$
• Learn $\pi _{\theta }:s_{t}->a_{t}$ to maximize $\sum _{t}r_{t}$