lstm_intro

11 / 15
LSTM v4: output projection layer
Reduces dimensionality of recursive messages Can speed up training without aecting results quality
mt −1
• • ◦ • • ◦
yt +1 = softmax (W ym mt ) mt = W mm (ot σ (ht )) ot = logistic W oe et + W om mt −1 + W oh ht + bo ht = fi ht −1 + it
σ W he et + W hm mt −1 + bh
it = logistic W ie et + W im mt −1 + W ih ht −1 + bi fi = logistic W fe et + W fm mt −1 + W fh ht −1 + bf et = W ex xt
9 / 15
LSTM v2: forget (remember) gate
Model controls when memory, ht , is reduced Forget gate should be called remember gate
mt −1
• • • •
bf
logistic
bo
ot
•
8 / 15
Constant Error Carousel
Allow the network to propagate errors without modication No nonlinearity in recursion
yt +1 = softmax (W y m mt ) mt = σ (ht ) ht = ht −1 +σ W he et + W hm mt −1 + bh et = W ex xt
10 / 15
LSTM v3: peepholes
Allow the gates to additionally see the internal memory state Diagonal matrices only (all others dense)
mt −1
• • • • ◦ ◦b
o
ot
logistic
6 / 15
Vanishing gradients
Gradients may vanish or explode Can aect any 'deep' network
e.g. ne-tuning a non-recurrent deep neural network
Image from Alex Graves' textbook
2 / 15
Context and notation
Model: yt +1 = p (xt | xt −1 , xt −2 , . . . , x1 ) for 1 ≤ t < L Minimize cross-entropy objective:
L−1
J=
t =1
H (yt +1 , xt ) =
8 / 15
Constant Error Carousel
Allow the network to propagate errors without modication No nonlinearity in recursion
yt mt −1 ht −1 et −1 xt −1 yt +1 mt ht et xt yt +2 mt +1 h t +1 et +1 xt +1
12 / 15
Gradients no longer vanish
Image from Alex Graves' textbook
13 / 15
LSTM implementations
RNNLIB (Alex Graves) /p/rnnl/ PyLearn2 (experimental code, in sandbox/rnn/models/rnn.py) Theano, e.g.
def lstm_step(x_t, m_tm1, h_tm1, w_xe, ..., b_o): e_t = dot(x_t, w_xe) i_t = sigmoid(dot(e_t, w_ei) + dot(m_tm1, w_mi) + c_tm1 * w_ci + b_i) f_t = sigmoid(dot(e_t, w_ef) + dot(m_tm1, w_mf) + c_tm1 * w_cf + b_f) h_t = f_t * h_tm1 + i_t * tanh(dot(e_t, w_eh) + dot(m_tm1, w_mh) + b_h) o_t = sigmoid(dot(e_t, w_eo) + dot(m_tm1, w_mo) + c_t * w_co + b_o) m_t = dot(o_t * tanh(h_t), w_mm) y_t = softmax(dot(m_t, w_my)) return m_t, c_t, y_t
yt ht et −3 xt −3 et −3 xt −2 et −3 xt −1
yt +1 = softmax W yh ht ht = σ W he [et −1 ; et −2 ; et −3 ] + bh et = W ex xt
4 / 15
Recurrent Neural Network (RNN)
8 / 15
Constant Error Carousel
Allow the network to propagate errors without modication No nonlinearity in recursion
mt −1 mt
ht −1
ht
et
• •
bh
dense matrix multiplication
yt +1 = softmax (W ym mt ) mt = ot σ (ht ) ot = logistic W oe et + W om mt −1 +W oh ht + bo ht = fi ht −1 + it
σ W he et + W hm mt −1 + bh
it = logistic W ie et + W im mt −1 +W ih ht −1 + bi fi = logistic W fe et + W fm mt −1 +W fh ht −1 + bf et = W ex xt
Attenuate input and output signals
yt +1 = softmax (W ym mt ) mt = ot σ (ht ) ot = logistic (W oe et + W om mt −1 + bo ) ht = ht −1 + it σ W he et + W hm mt −1 + bh it = logistic W ie et + W im mt −1 + bi et = W ex xt
Error gradients pass through nonlinearity every step
Image from
https://
Unless weights large, error signal will degrade
δh = σ (•) W (h+1)h δh+1
7 / 15
Constant Error Carousel
Allow the network to propagate errors without modication No nonlinearity in recursion
yt mt −1 ht −1 et −1 xt −1 yt +1 mt ht et xt yt +2 mt +1 h t +1 et +1 xt +1
bo
ot
logistic
•
mt
bf
ft
logistic
ht −1
• • ◦
bi
ht it
logistic
et
• •
bh
12 / 15
LSTM v4: output projection layer
Reduces dimensionality of recursive messages Can speed up training without aecting results quality
yt +1 = softmax (W ym mt ) mt = ot σ (ht ) ot = logistic (W oe et + W om mt −1 + bo ) ht = fi ht −1 + it
σ W he et + W hm mt −1 + bh
it = logistic W ie et + W im mt −1 + bi fi = logistic W fe et + W fm mt −1 + bf et = W ex xt
14 / 15
Further thoughts
Sequences vs. hierarchies vs. plain 'deep' Other solutions to vanishing gradients