神经网络之——递归神经网络

Continuous time computing
Lecture 10 Total 69 pages
23
From Discrete Computing to Continuous Computing
Changing time steps
Lecture 10 Total 69 pages
24
From Discrete Computing to Continuous Computing
34
Trajectories
da(t) ga(t), p(t),t
dt
If a1 a2, then
at, a1 at, a2 for any t 0
Lecture 10 Total 69 pages
35
Trajectories
da(t) ga(t), p(t),t
dt
If a1 a2, then
Lecture 10 Total 69 pages
27
From Discrete Computing to Continuous Computing
x(t ) x(t) x(t) f wx(t) b
x(t ) x(t) x(t) f wx(t) b
Lecture 10 Total 69 pages
43
Linear RNNs
a(t) a t
da(t) a(t) Wa(t) p
dt
Lecture 10 Total 69 pages
44
Linear RNNs
a(t) a t
•
a1 (t )
•
a1 (t )
1a2
(t)
p1
a2 (t) a2 (t) 2a1(t) p2
x2 (k 1) f2 w21x1(k) w22x2 (k)
f2
n2 w21x1(k) w22x2(k)
w12 x2 (k )
w22 x2 (k )
w11 x1(k )
w21
x1(k )
Lecture 10 Total 69 pages
13
x1(k 1)
f1 n1
w12 x2 (k ) w11 x1(k )
33
Trajectories
da(t) ga(t), p(t),t
dt given any initial condition a(0), thereis a trajectory a(t, a(0))
Trajectories space at, a0 t 0, a0Rn
Lecture 10 Total 69 pages
31
Recurrent NNs
Network time
da(t) ga(t), p(t),t
dt
Network state
Network input
What’s the output of a RNN?
a(t) a t
Network output
Lecture 10 Total 69 pages
da(t) ga(t), p(t),t
dt
Does each trajectory of a RNN converge to an equilibrium?
Methods: 1. Solving differential equation directly;
2. Energy method.
Lecture 10 Total 69 pages
外核层
内核层
• 三层神经网络：神经节细胞层-内核层-外核层 • 每层内各神经元之间无连接 • 前一层神经元计算完后传递给下一层神经元进行计算
Lecture 10 Total 69 pages
4
Feedforward NNs
x1 w11
w12 w13
w21
x2
w22
w23
wR1 wR 2
wR3 xR
x(t 1) x(t) 1 x(t) f wx(t) b
Lecture 10 Total 69 pages
26
From Discrete Computing to Continuous Computing
x(t 1) x(t) 1 x(t) f wx(t) b
x(t ) x(t) x(t) f wx(t) b
da(t) ga(t), p(t),t
dt
Equilibrium point: g a, p(t), t 0 for all t 0
Lecture 10 Total 69 pages
38
Equilibrium Points
da(t) a(t) p dt
a(t) a(0)et p 1 et
x2 (k 1)
f2
n2 w21x1(k) w22x2(k)
w12 x2 (k )
w22 x2 (k )
w11 x1(k )
w21
x1(k )
Lecture 10 Total 69 pages
12
Recurrent NNs
x1(k 1) f1(w11x1(k) w12x2 (k))
f1
n1 w11x1(k) w12x(k)
Recurrent Neural Networks
Lecture 10 Total 69 pages
1
Classification of NNs
Neural Networks
Feedforward NNs
Recurrent NNs
Lecture 10 Total 69 pages
2
视网膜信息处理的基本系统
at, a1 at, a2 for any t 0
Lecture 10 Total 69 pages
36
A Simple Example
da(t) a(t) p dt
a(t) a(0)et p 1 et
Lecture 10 Total 69 pages
37
Equilibrium Points
x(k 1) f wx(k) b
Network computing
x0
Input
x
RNN
Output
x3 x4 x2
x5
x1
x
x6
x0
x8 x7
Lecture 10 Total 69 pages
20
Computing:Discrete or Continuous ?
Lecture 10 Total 69 pages
x(t 1) f wx(t) b
x(t 1) x(t) x(t) f wx(t) b
Lecture 10 Total 69 pages
25
From Discrete Computing to Continuous Computing
x(t 1) x(t) x(t) f wx(t) b
Recurrent NNs
x1(k 1) f1w11x1(k) w12x2 (k)
x2 (k 1) f2 w21x1(k) w22x2 (k)
x2 (k 1)
f2 n2
w22 x2 (k )
w21
x1(k )
Lecture 10 Total 69 pages
14
Recurrent NNs
dx(t) x(t) f wx(t) b
dt
Lecture 10 Total 69 pages
30
Recurrent NNs
RNN model:
da(t) ga(t), p(t),t
dt
Network time
Network state
Network input
Lecture 10 Total 69 pages
10
Recurrent NNs
x1(k 1)
f1 n1
x2 (k 1)
f2 n2
w12 x2 (k )
w22 x2 (k )
w11 x1(k )
w21
x1(k )
Lecture 10 Total 69 pages
11
Recurrent NNs
x1(k 1)
f1
n1 w11x1(k ) w12 x2 (k )
Lecture 10 Total 69 pages
18
Discrete Time RNNs
x(k 1) f wx(k) b
Network computing ?
x3 x4 x2
x5
x1
x
x6
x0
x8 x7
Lecture 10 Total 69 pages
19
Discrete Time RNNs
x1(k 1) f1w11x1(k) w12x2 (k)
x2 (k 1) f2 w21x1(k) w22x2 (k)
n
xi (k 1) fi wij x j (k)
j1
x1(k 1)
f1 n1
x2 (k 1)
f2 n2
w12 x2 (k ) w11 x1(k )
w22 x2 (k ) w21 x1(k )
41
Method One
Solving Differential Equations
Lecture 10 Total 69 pages
42
A Simple Example
da(t) a(t) p dt
a
a(t) a(0)et p 1 et
p
t
Lecture 10 Total 69 pages
32
Convergence of RNNs
da(t) ga(t), p(t),t
dt
Network state
a(t) a t Converge ?
Equilibrium point: g a, p(t), t 0 for all t 0
Lecture 10 Total 69 pages
Lecture 10 Total 69 pages
15
Recurrent NNs
x(k 1) f wx(k)
Lecture 10 Total 69 pages
16
Recurrent NNs
b
b
b
b
Lecture 10 Total 69 pages
17
Discrete Time RNNs
x(k 1) f wx(k) b
视网膜分3层神经细胞（自下而上）： 外层、中间层、最后层
光信息自光感受器经双极细胞传至 神经节细胞，神经节细胞的轴突汇 聚成视神经离开眼球。
水平细胞和无长突细胞通过侧向联系调 节双极细胞和神经节细胞的反应。
Lecture 10 Total 69 pages
3
Feedforward NNs
神经节细胞层
Lecture 10 Total 69 pages
8
Recurrent NNs
f1
f2
n1
n2
Lecture 10 Total 69 pages
9
Recurrent NNs
x1(k Байду номын сангаас)
f1 n1
x2 (k 1)
f2 n2
x2 (k)
x2 (k)
x1(k )
x1(k )
Lecture 10 Total 69 pages
Lecture 10 Total 69 pages
45
http://hebb.mit.edu/people/seung/index.html
n f 1 1 a11 11
n f1
1
a
1 2
22
n f2 2 a12 11
n f2
2
a
2 2
22
n f 1 s1
a1
1 s1 s1
n f 2
2
a2 s2
s2 s2
a f Wx
n f 3 3 a12 11
n f 3
3
a
2 2
22
n f 3 s3
3 s3
a2 s2
Lecture 10 Total 69 pages
28
From Discrete Computing to Continuous Computing
x(t ) x(t) x(t) f wx(t) b
0
dx(t) x(t) f wx(t) b
dt
Lecture 10 Total 69 pages
29
Continuous Computing RNNs
5
Recurrent NNs
Contain feedback among neurons
Lecture 10 Total 69 pages
6
Recurrent NNs
Lecture 10 Total 69 pages
7
Recurrent NNs
How to derive math models of RNNs?
21
Discrete vs Continuous
Discrete time computing
Continuous time computing
Lecture 10 Total 69 pages
22
Discrete vs Continuous
How to derive continuous time computing math models of RNNs?
a
a p
p
t
Lecture 10 Total 69 pages
39
Convergence of RNNs
a(t) a t
da(t) ga(t), p(t),t
dt
Attractors
Lecture 10 Total 69 pages
40
Convergence of RNNs
a(t) a t