统计学习[The Elements of Statistical Learning]第八章习题

The Element of Statistical Learning –Chapter 8

oxstar@SJTU

January 6,2011

Ex.8.1Let r (y )and q (y )be probability density functions.Jensen’s inequality states that for a random variable X and a convex function φ(x ),E[φ(X )]≥φ[E(X )].Use Jensen’s inequality to show that

E q log[r (Y )/q (Y )]

is maximized as a function of r (y )when r (y )=q (y ).Hence show that R (θ,θ)≥R (θ ,θ)as stated below equation (8.46).

Proof .−log(x )is a convex function,so from Jensen’s inequality we have

E q −log[r (Y )/q (Y )]≥−log[E q (r (Y )/q (Y ))]=−log[ r (y )q (y )q (y )d y ]=−log[ r (y )d y ]

=−log 1=0

In other words,E

q log[r (Y )/q (Y )]≤0,and iff.r (y )=q (y )it get its maximum.

R (θ ,θ)−R (θ,θ)=E[ 1(θ ;Z m |Z )|Z ,θ]−E[ 1(θ;Z m |Z )|Z ,θ]

=E Pr(Z m |Z ,θ)[log Pr(Z m |Z ,θ )]−E Pr(Z m |Z ,θ)[log Pr(Z m |Z ,θ)]

=E Pr(Z m |Z ,θ)(log Pr(Z m |Z ,θ )Pr(Z m |Z ,θ)

)≤0Hence we have R (θ,θ)≥R (θ ,θ)and iff.Pr(Z m |Z ,θ )=Pr(Z m |Z ,θ),the equation is satisfied.

Ex.8.4Consider the bagging method of Section 8.7.Let our estimate ˆf

(x )be the B -spline smoother ˆµ(x )of Section 8.2.1.Consider the parametric bootstrap of equation (8.6),applied to this estimator.Show that if we bag ˆf (x ),using the parametric bootstrap to generate the bootstrap samples,the bagging estimate ˆf bag (x )converges to the original estimate ˆf (x )as B →∞.Proof .According to the definition of bagging estimate

ˆf bag (x )=1B B b =1

ˆf ∗b (x )=ˆf ∗(x )we have

E(ˆf bag (x ))=E(ˆf ∗(x ))=E(ˆf ∗(x ))

(1)Var(ˆf bag (x ))=Var(ˆf ∗(x ))=Var(ˆf ∗(x ))B (2)

The function estimated from a parametric bootstrap sample isˆf∗(x)and it has distribution ˆf∗(x)∼N(ˆf(x),h(x)T(H T H)−1h(x)ˆσ2)=N(ˆf(x),ˆσ∗2)

Hence we have

E(ˆf∗(x))=ˆf(x)(3)

Var(ˆf∗(x))=ˆσ∗2(4)

lim B→∞Var(ˆf bag(x))=lim

B→∞

ˆσ∗2

B

=0//from(1)-(4)

so the bagging estimateˆf

bag

(x)converges to its expected value,i.e.,the original estimateˆf(x)