强化学习基本概念与训练算法

Policy update improves the Q-value
Convergence Condition of Monte Carlo
1. Every state-action pair should be visited. 2. Infinite policy evaluation and update iterations.
Monte Carlo with Exploring Starts (MC-ES)
Figure borrowed from ”Reinforcement Learning: An Introduction ” by Sutton et al.
Exploration and Exploitation
Deterministic Policy：
Stochastic Policy：
Search Space：
Utility/Value Function
Value：The expected return that the agent starts from a certain state s.
state-value function：
Basic Conceptions of RL
Environment: 1. provide the reward signals to the agent 2. is part of the state
Figure borrowed from Yu Yang(NJU)
Basic Conceptions of RL
Learning Algorithm
Policy gradient Value Function (Value Iteration, Q-Learning, Sarsa, etc.)
Actor-Critic Policy Search
Bellman(Dynamic) Equation
Optimal Policy
Markov Decision Process
s
…
…
…
Markov Decision Process(MDP)
MDP Solutions
--Linear Programming
MDP Solutions
--Policy Iteration(PI)
Figure borrowed from ”Reinforcement Learning: An Introduction ” by Sutton et al.
Return：all the rewards.

Finite steps return：
Infinite steps return：
is the discount factor.
Average return：
Policy Function
Policy：The agent’s decision strategy (stable and stochastic).
s a s’
Model-free：Reward and the State Transition Probability are not modeled (agent do not care internal mechanism of the environment ).
Markov Decision Process
1. Markov Property: The next state is only depend on the current state(no-memory). 2. A model-based model: Need environment modeling (states transition matrix and reward function)
is a episode start from (s, a)
MC Policy Evaluation
Value function is the average return of many episode.
…
FVMC(first visit MC) EVMC(every visit MC)
Figure borrowed from https:///jinxulin/p/3560737.html
1. 2. 3. 4.
States Actions Rewards Policy
Figure borrowed from Yu Yang(NJU)
Agent Decision Process
s0
r1 a0
s1
r2 a1
s2
r3 a2
…
st
rt+1 at
st+1
at+1
rt+2 …
Reward & Return
MDP Solutions
Figure borrowed from ”Reinforcement Learning: An Introduction ” by Sutton et al.
Monte Carlo RL
1. A model-free policy iteration method. 2. Iteratively estimate the action-state-value function and the policy function .
MC Policy Improvement
The best action a for state s is the one that has maximum Q(s,a)
MC RL Algorithm Framework
Policy estimation:
Policy update:
MC RL Convergence Proof
Learning Methods
• Supervised Learning：explicit signals
1. The training data is <instance, label>. 2. The general objective is minimizing the gap between “prediction” and “label”. 3. Most of the labels are from manual work.
Agent: 1. take an action at each step conditioned on the current state. 2. After the action, agent gets the reward, and the state changes.
Figure borrowed from Yu Yang(NJU)
MC Policy Evaluation
MC Policy Improvement
MC Policy Evaluation
Q(s, a) function is the average return of many episodes start from (s,a).
Discount factor
Learning Methods
• Unsupervised Learning：no signals
1. The training data is <instance>. 2. The general objective is finding some potential patterns under some heuristic criterions.
state-action-value function：
GridWorld
Figure borrowed from /rlpart3.html
GridWorld
=Solved!
=Solved!
=Solved!
Figure borrowed from /rlpart3.html
Reinforcement Learning
--basic conceptions
By ZJC 2018.05
Learning Methods
• Supervised Learning：explicit signals • Unsupervised Learning：no signals • Semi-Supervised Learning：partial/weak signals
Theorem 1:
Theorem 2:
Theorem 3:
Theorem 4:
Model-free or Model-Based
Environment: 1. provide the reward signals to the agent 2. is part of the state Model-based：Reward and the State Transition Probability are modeled.
Goal of the agent: take right actions(policy) to achieve the maximum long-term total rewards(return).
Figure borrowed from Yu Yang(NJU)
Formulation of RL
--Generalized Iteration(GI) 1. In policy iteration, these two processes alternate, each completing before the other begins, but this is not really necessary. 2. Let policy evaluation and policy improvement processes interact, independent of the granularity and other details of the two processes.