机器学习基石资源-01_Motivation_and_Primal_Problem_14-27

L(b, w, ξ , α, β ) = 1 T w w+C· 2
N N
ξn
n=1 N
+
n=1
αn · 1 − ξn − yn (wT zn + b) +
n =1
βn · (−ξn )
want: Lagrange dual
αn ≥0, βn ≥0
Hsuan-Tien Lin (NTU CSIE)
max
Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 3/22
yn (w zn + b) ≥ 1 for correct n
Soft-Margin Support Vector Machine
Motivation and Primal Problem
yn = sign(wT zn + b)
n =1
min
b,w
min
b,w
1 T w w 2 yn (wT zn + b) ≥ 1 for all n
s.t.
combination:
min
b ,w
1 T w w+C· 2
T
N
yn = sign(wT zn + b)
n=1
s.t.
yn (wT zn + b) ≥ −∞ for incorrect n C : trade-off of large margin & noise tolerance
and explicit constraint 0 ≤ αn ≤ C : βn removed
ξ can also be removed :-), like how we removed b
max min 1 T w w+ 2
N
0≤αn ≤C , βn =C −αn
b,w,ξ
n=1
αn (1 − yn (wT zn + b))
Lecture 4: Soft-Margin Support Vector Machine
Motivation and Primal Problem Dual Problem Messages behind Soft-Margin SVM Model Selection
2 3
Combining Predictive Features: Aggregation Models Distilling Implicit Features: Extraction Models
National Taiwan University (國立台灣大學資訊工程系)
Hsuan-Tien Lin (NTU CSIE)
Machine Learning Techniques
0/22
Soft-Margin Support Vector Machine
Roadmap
1
Embedding Numerous Features: Kernel Models
Soft-Margin SVM (1/2)
min
b,w
1 T w w+C· 2
N
yn = sign(wT zn + b)
n=1
s.t.
yn (wT zn + b) ≥ 1 − ∞ · yn = sign(wT zn + b)
• · : non-linear, not QP anymore :-(
—what about dual? kernel?
Machine Learning Techniques 4/22
Soft-Margin Support Vector Machine
Motivation and Primal Problem
Soft-Margin SVM (2/2)
Hi
• penalize with margin violation
˜ + 1 + N variables, 2N constraints • QP of d ˜ by next: remove dependence on d soft-margin SVM primal ⇒ dual?
Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 5/22
1 2 3 4
1
11 21 31
Reference Answer: 2
ξ1 is simply 1 − y1 (wT z1 + b) when y1 (wT z1 + b) ≤ 1.
Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 6/22
Soft-Margin Support Vector Machine
• part of reasons: Φ
• other part: separable
Φ1 if always insisting on separable (=⇒ shatter), have power to overfit to noise
Φ4
Hsuan-Tien Lin (NTU CSIE)
Machine Learning Techniques
N
N
ξn
n=1 N
+
n=1
αn · 1 − ξn − yn (wT zn + b) +
n=1
βn · (−ξn )
∂L • ∂ξ = 0 = C −αn −βn n
• no loss of optimality if solving with implicit constraint βn = C − αn
• penalize with margin violation instead of error count
—quadratic objective
b,w,ξ
soft-margin SVM: min
1 T w w+C· 2
N
ξn
n=1
s.t.
Hsuan-Tien Lin (NTU CSIE)
பைடு நூலகம்
yn (wT zn + b) ≥ 1 − ξn and ξn ≥ 0 for all n
b,w,ξ
• record ‘margin violation’ by ξn 1 T w w+C· 2
N
min
ξn
n =1
violation
s.t.
yn (wT zn + b) ≥ 1 − ξn and ξn ≥ 0 for all n
Hi
• parameter C : trade-off of large margin & margin violation • large C : want less margin violation • small C : want large margin
XX NX XXX + (C − α ) · ξn n − n Xβ XXX =1 n
Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 8/22
Soft-Margin Support Vector Machine
Dual Problem
Other Simplifications
max min
b,w
0≤αn ≤C , βn =C −αn
1 T w w+ 2
N
n=1
αn (1 − yn (wT zn + b))
familiar? :-)
• inner problem same as hard-margin SVM
Soft-Margin Support Vector Machine
Motivation and Primal Problem
Fun Time
At the optimal solution of min 1 T w w+C· 2
N
b,w,ξ
ξn
n=1
s.t.
yn (wT zn + b) ≥ 1 − ξn and ξn ≥ 0 for all n,
Machine Learning Techniques 1/22
Hsuan-Tien Lin (NTU CSIE)
Soft-Margin Support Vector Machine
Motivation and Primal Problem
Cons of Hard-Margin SVM
recall: SVM can still overfit :-(
Dual Problem
Lagrange Dual
primal:
b ,w ,ξ
min
1 T w w+C· 2
N
ξn
n=1
s.t.
yn (wT zn + b) ≥ 1 − ξn and ξn ≥ 0 for all n
Lagrange function with Lagrange multipliers αn and βn
standard dual can be derived using the same steps as Lecture 2
Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 9/22
Soft-Margin Support Vector Machine
assume that y1 (wT z1 + b) = −10. What is the corresponding ξ1 ?