Reinforcement Learning Memo

A simple memo for learning Reinforcement Learning.（强化学习备忘录）

Terminologies

1. State 状态 $s$, Action 动作 $a$, Reward 奖励 $r$
2. Policy 策略 $\pi: (s,a) \to [0, 1]$

   $$\pi(s | a) = \mathbb{P}(A = a | S = s)$$

3. Trajectory 轨迹: $s_1, a_1, r_1, s_2, a_2, r_2, \dots, s_T, a_T, r_T$
4. Discounted return 折扣回报

   $$U_t = R_t + \gamma R_{t + 1} + \gamma^2 R_{t + 2} + \gamma^3 R_{t + 3} + \cdots$$

5. Action-value function 动作价值函数

   $$Q_\pi(s_t,a_t) = \mathbb{E}[U_t | S = s_t, A = a_t]$$

6. Optimal action-value function 最优动作价值函数

   $$Q^*(s_t,a_t) = \max_\pi Q_\pi(s_t,a_t)$$

7. State-value function 状态价值函数

   $$V_\pi(s_t) = \mathbb{E}_A[Q_\pi(s_t, A)] = \sum_{a \in A} \pi(a|s_t) \cdot Q_\pi(s_t, a)$$

   • $\mathbb{E}_S[V_\pi(S)]$ evaluates how good the policy $\pi$ is.

Overview

References

1. 王树森,黎或君,张志华.深度强化学习.人民邮电出版社,2022.https://github.com/wangshusen/DRL
2. 张伟楠,沈键,俞勇.动手学强化学习.人民邮电出版社,2022.https://github.com/boyu-ai/Hands-on-RL
3. https://www.bilibili.com/video/BV1rooaYVEk8
4. https://www.bilibili.com/video/BV15cZYYvEhz