统计学习[The Elements of Statistical Learning]第四章习题

2
ˆ = (N − 2)Σ
k=1 gi =k 2
(xi − µ ˆk )(xi − µ ˆk )T (xi xT ˆT ˆk µ ˆT i − 2 xi µ k +µ k)
k=1 gi =k 2
(9)
=
// xT ˆ k = xi µ ˆT i µ k
(10)
=
k=1 N gi =k
xi xT ˆk µ ˆT i − Nk µ k
(5)
=
k=1
Nk µ ˆk
N yk − (N1 y1 + N2 y2 ) N
x i = Nk µ ˆk
(6) (7) (8)
N1 N2 (y2 − y1 )(ˆ µ2 − µ ˆ1 ) N = N (ˆ µ2 − µ ˆ1 ) // y1 = N/N1 , y2 = N/N2 =
2
We also have
//
gi =k
x i = Nk µ ˆk
(11)
=
i=1 N
xi xT ˆ1 µ ˆT ˆ2 µ ˆT i − (N1 µ 1 + N2 µ 2)
2
(12)
xi x ¯T =
i=1 k=1 gi =k
xk
2 k=1
gi =k
xk
T
N
(13) (14)
= Meanwhile
N N
1 (N1 µ ˆ 1 + N2 µ ˆ2 )(N1 µ ˆ 1 + N2 µ ˆ2 )T N
1
ˆ0 , and hence the predicted values f ˆ= β ˆ0 + β ˆT x. Consider the following 5. Find the solution β rule: classify to class 2 if y ˆi > 0 and class 1 otherwise. Show this is not the same as the LDA rule unless the classes have equal numbers of observations. (Fisher, 1936; Ripley, 1996) Proof 1. Consider the log-ratio of each class density (equation 4.9 in text book) log π2 1 Pr(G = 2|X = x) = log − (µ2 + µ1 )T Σ−1 (µ2 − µ1 ) + xT Σ−1 (µ2 + µ1 ) Pr(G = 1|X = x) π1 2
N1 N2 N (y2
(17) (18)
(19) (20) (21)
4. By replacing N with ˆ= β
− y1 ) (from (7) and (8)) and from (20), we still have
N1 N2 ˆ −1 (ˆ ˆ −1 (ˆ (y2 − y1 ) − (ˆ µ2 − µ ˆ1 )T β Σ µ2 − µ ˆ1 ) ∝ Σ µ2 − µ ˆ1 ) N (N − 2)
When it > 0, the LDA rule will classify x to class 2, meanwhile, we need to estimate the parameters of the Gaussian distributions using our training data 1 ˆ −1 (ˆ ˆ −1 (ˆ µ2 + µ ˆ 1 )T Σ µ2 − µ ˆ1 ) + log(π1 ) − log(π2 ) xT Σ µ2 − µ ˆ1 ) > (ˆ 2 1 T ˆ −1 1 T ˆ −1 N1 N2 = µ ˆ Σ µ ˆ2 − µ ˆ Σ µ ˆ1 + log − log 2 2 2 1 N N 2. Let β = (β, β0 )T and compute the partial deviation of the RSS(β ), then we have ∂ RSS(β ) = −2 (yi − β0 − β T xi ) = 0 ∂β0 i=1 ∂ RSS(β ) = −2 xi (yi − β0 − β T xi ) = 0 ∂β i=1 We can also derive that β0 =
1
a = W− 2 b,
1
aT = bT W− 2
1
Hence the generalized eigenvalue problem max(aT Ba) = max(bT W− 2 BW− 2 b)
a b
1 1
subject to
aT Wa = bT W− 2 WW− 2 b = 1 So the problem is transformed to a standard eigenvalue problem. Ex. 4.2 Suppose we have features x ∈ Rp , a two-class response, with class sizes N1 , N2 , and the target coded as −N/N1 , N/N2 . 1. Show that the LDA rule classies to class 2 if 1 T ˆ −1 N1 1 T ˆ −1 N2 ˆ −1 (ˆ ˆ2 Σ µ ˆ2 − µ ˆ1 Σ µ ˆ1 + log xT Σ µ2 − µ ˆ1 ) > µ − log , 2 2 N N and class 1 otherwise. 2. Consider minimization of the least squares criterion
N
N
xi [β (xi − x ¯)] =
i=1 i=1
T
xi [(xi − x ¯) β ] =
i=1
T
xi xT i
−
i=1
xi x ¯T β
(15)
1 ˆ + (N1 µ = (N − 2)Σ ˆ1 µ ˆT ˆ2 µ ˆT (N1 µ ˆ 1 + N2 µ ˆ2 )(N1 µ ˆ 1 + N2 µ ˆ2 )T β // from (12)(14) 1 + N2 µ 2)− N (16) ˆ + N1 N2 (ˆ µ2 µ ˆT ˆ1 µ ˆT ˆ2 µ ˆT ˆ1 µ ˆT = (N − 2)Σ 2 −µ 2 −µ 1 +µ 1) β N ˆ B β = N (ˆ ˆ + N1 N2 Σ µ2 − µ ˆ1 ) // from (8) = (N − 2)Σ N 3. ˆ B β = (ˆ Σ µ2 − µ ˆ1 )(ˆ µ2 − µ ˆ 1 )T β ˆ B β is in the direction (ˆ (ˆ µ2 − µ ˆ1 )T β is a scalar, therefore Σ µ2 − µ ˆ1 ), and ˆ= β N1 N2 ˆ 1 ˆ −1 Σ N (ˆ µ2 − µ ˆ1 ) − ΣB β // from (18) N −2 N 1 N1 N2 ˆ −1 (ˆ = N− (ˆ µ2 − µ ˆ 1 )T β Σ µ2 − µ ˆ1 ) N −2 N ˆ −1 (ˆ ∝Σ µ2 − µ ˆ1 )
The Element of Statistical Learning – Chapter 4
oxstar@SJTU January 6, 2011
Ex. 4.1 Show how to solve the generalized eigenvalue problem max aT Ba subject to aT Wa = 1 by transforming to a standard eigenvalue problem. Answer W is the common covariance matrix, and it’s positive-semidefinite, so we can define b = W 2 a,
ˆ µ ˆy k = ˆ µ ˆy = gi =k
y ˆi y ˆi
Nk
gi =
= =
gi =k
ˆ T xi B
gi =
Ni ˆ T xi B
ˆTµ =B ˆx k ˆTµ =B ˆx
N N N(1)(2) Nhomakorabea1 N
N
N
(yi − β T xi )
i=1
// from (1)
(3)
xi [β T (xi − x ¯)] =
i=1 i=1 2
xi yi −
1 N
N
yj
j =1
// from (2)(3)
(4)
=
k=1 gi =k 2
xi yk −
N1 y1 + N2 y2 N //
gi =k
Compare (24) with (25), we can find that they are the same rule. But when N1 = N2 , these rules are obviously different. ˆ via linear regression. In detail, let Ex. 4.3 Suppose we transform the original predictors X to Y T −1 T ˆ ˆ Y = X(X X) X Y = XB, where Y is the indicator response matrix. Similarly for any input ˆ T x ∈ RK . Show that LDA using Y ˆ is identical to LDA x ∈ Rp , we get a transformed vector y ˆ=B in the original space. ˆ T x, so we have Proof Transformed vector y ˆ=B