(完整版)国科大中科院人工智能与机器学习12-DL
合集下载
相关主题
- 1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
- 2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
- 3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。
年NN研究的低潮
6
2nd Generation Neural Networks
Institute of Computing Technology, Chinese Academy of Sciences
多层感知机(Multi-layer Perceptron, MLP)
超过1层的hidden layers（正确输出未知的层）
5
Institute of Computing Technology, Chinese Academy of Sciences
第一代神经网络
单层感知机(Perceptrons)模型的局限性
Minsky & Papert的专著Perceptron(1969) 只能对线性可分的模式进行分类 解决不了异或问题 几乎宣判了这类模型的死刑，导致了随后多
Ʃ ������ ������1
������3
Ʃ ������
Ʃ ������ ������2
Ʃ ������
David E. Rumelhart,, Geoffrey E. Hinton, and Ronald J. Williams. (Oct.1986). "Learning
representations by back-propagating errors". Nature 323 (6088): 533–536
Institute of Computing Technology, Chinese Academy of Sciences
Frank Rosenblatt(1957), The Perceptron--a perceiving and recognizing
automaton. Report 85-460-1, Cornell Aeronautical Laboratory.
关于DL的思考与讨论
2
机器学习的基本任务
Institute of Computing Technology, Chinese Academy of Sciences
x
F ( x)
Class label (Classification)
y
Vector (Estimation)
Object recognition
������������ = ������ ������������
f为激活函数，其中：
������
������
������������ = ������������������������ − ������ = ������������������������
������=1
������=0
7
Error Backpropagation
W is the parameter of the network; J is the objective function
Institute of Computing Technology, Chinese Academy of Sciences
Target values
神经元之间通过突触(synapse)连接
层级感受野，学习使突触连接增强或变弱甚至消失
Hubel, D. H. & Wiesel, T. N. (1962)
4
第一代神经网络
感知机(Perceptrons)模型[Rosenblatt, 1957]
二类分Βιβλιοθήκη Baidu，单个神经元的功能(输入输出关系)
Almost all data is unlabeled.
The learning time does not scale well
It is very slow in networks with multiple hidden layers.
It can get stuck in poor local optima
深度学习：快速推进中的 机器学习与人工智能前沿
山世光 中科院计算所
Institute of Computing Technology, Chinese Academy of Sciences
提纲
深度学习(DL)及其应用前沿 DL在CV领域应用的启示 关键算法介绍
Perceptron及学习算法 MLP及其BP算法 Auto-Encoder CNN及其主要变种
Feedforward operation
Output layer Hidden layers
Back error propagation
Input layer
David E. Rumelhart,, Geoffrey E. Hinton, and Ronald J. Williams. (Oct.1986). "Learning representations by back-propagating errors". Nature 323 (6088): 533–536
{dog, cat, horse,, …}
Low-resolution image
Super resolution
High-resolution image
3
Institute of Computing Technology, Chinese Academy of Sciences
源起——生物神经系统的启示
Institute of Computing Technology, Chinese Academy of Sciences
2nd Generation Neural Networks
理论上多层好
两层权重即可逼近任何连续函数映射
遗憾的是，训练困难
It requires labeled training data
BP算法 [Rumelhart et al., 1986]
Compute error signal; Then, back-propagate error signal to get
derivatives for learning
������1 ������2
Ʃ ������
Ʃ ������
These are often quite good, but for deep nets they are far from optimal.
6
2nd Generation Neural Networks
Institute of Computing Technology, Chinese Academy of Sciences
多层感知机(Multi-layer Perceptron, MLP)
超过1层的hidden layers（正确输出未知的层）
5
Institute of Computing Technology, Chinese Academy of Sciences
第一代神经网络
单层感知机(Perceptrons)模型的局限性
Minsky & Papert的专著Perceptron(1969) 只能对线性可分的模式进行分类 解决不了异或问题 几乎宣判了这类模型的死刑，导致了随后多
Ʃ ������ ������1
������3
Ʃ ������
Ʃ ������ ������2
Ʃ ������
David E. Rumelhart,, Geoffrey E. Hinton, and Ronald J. Williams. (Oct.1986). "Learning
representations by back-propagating errors". Nature 323 (6088): 533–536
Institute of Computing Technology, Chinese Academy of Sciences
Frank Rosenblatt(1957), The Perceptron--a perceiving and recognizing
automaton. Report 85-460-1, Cornell Aeronautical Laboratory.
关于DL的思考与讨论
2
机器学习的基本任务
Institute of Computing Technology, Chinese Academy of Sciences
x
F ( x)
Class label (Classification)
y
Vector (Estimation)
Object recognition
������������ = ������ ������������
f为激活函数，其中：
������
������
������������ = ������������������������ − ������ = ������������������������
������=1
������=0
7
Error Backpropagation
W is the parameter of the network; J is the objective function
Institute of Computing Technology, Chinese Academy of Sciences
Target values
神经元之间通过突触(synapse)连接
层级感受野，学习使突触连接增强或变弱甚至消失
Hubel, D. H. & Wiesel, T. N. (1962)
4
第一代神经网络
感知机(Perceptrons)模型[Rosenblatt, 1957]
二类分Βιβλιοθήκη Baidu，单个神经元的功能(输入输出关系)
Almost all data is unlabeled.
The learning time does not scale well
It is very slow in networks with multiple hidden layers.
It can get stuck in poor local optima
深度学习：快速推进中的 机器学习与人工智能前沿
山世光 中科院计算所
Institute of Computing Technology, Chinese Academy of Sciences
提纲
深度学习(DL)及其应用前沿 DL在CV领域应用的启示 关键算法介绍
Perceptron及学习算法 MLP及其BP算法 Auto-Encoder CNN及其主要变种
Feedforward operation
Output layer Hidden layers
Back error propagation
Input layer
David E. Rumelhart,, Geoffrey E. Hinton, and Ronald J. Williams. (Oct.1986). "Learning representations by back-propagating errors". Nature 323 (6088): 533–536
{dog, cat, horse,, …}
Low-resolution image
Super resolution
High-resolution image
3
Institute of Computing Technology, Chinese Academy of Sciences
源起——生物神经系统的启示
Institute of Computing Technology, Chinese Academy of Sciences
2nd Generation Neural Networks
理论上多层好
两层权重即可逼近任何连续函数映射
遗憾的是，训练困难
It requires labeled training data
BP算法 [Rumelhart et al., 1986]
Compute error signal; Then, back-propagate error signal to get
derivatives for learning
������1 ������2
Ʃ ������
Ʃ ������
These are often quite good, but for deep nets they are far from optimal.