Open main menu

Humanoid Robots Wiki β

Changes

Allen's REINFORCE notes

141 bytes removed, 25 May
no edit summary
=== Motivation ===
Recall that the objective of Reinforcement Learning is to find an optimal policy <math> \pi^* </math> which we encode in a neural network with parameters <math>\theta^*</math>. <math> \pi_\theta </math> is a mapping from observations to actions. These optimal parameters are defined as
<math>\theta^* = \text{argmax}_\theta E_{\tau \sim p_\theta(\tau)} \left[ \sum_t r(s_t, a_t) \right] </math>. Let's unpack what this means. To phrase it in english, this is basically saying that the optimal policy is one such that the expected value of the total reward over following a trajectory (<math> \tau </math>) determined by the policy is the highest over all policies.
=== Overview ===
# ‎<syntaxhighlight lang="bash" line>Initialize neural network with input dimensions = observation dimensions and output dimensions = action dimensions. Remember a policy is a mapping from observations to outputs. If the space is continuous, it may make more sense to make output be one mean and one standard deviation for each component of the action. ‎<syntaxhighlight lang="python" line># For # of episodes:## While not terminated:### Get observation from environment### Use policy network to map observation to action distribution### Randomly sample one action from action distribution### Compute logarithmic probability of that action occurring### Step environment using action and store reward## Calculate loss over entire trajectory as function of probabilities and rewards
</syntaxhighlight>
=== Loss Function ===
53
edits