Lecture 3 Penalization and TGDR(1)

7
Lasso
8
Lasso pen ( β ) = λn ∑i =1 | β i |
p
• L1 penalty also has a long history in statistics • Credit of Lasso goes to R. Tibshirani (1996; Stanford Statistics) • Tuning parameter can be selected via cross validation • L1 penalty is equivalent to a L1 constraint (we know there is a one-one correspondence; we cannot write it down though)
2
Simple Solutions
• Pre-screening: select only a few (say 10) and analyze using existing approaches (we will come back to this later) • Step-wise, best subset: unstable. Also computational problems.
30
Variable selection with U-estimates
• So far, we have not said much about the models and objective functions. • The likelihood needs to be built on specific assumptions on data generating models • When there is suspicion on the model assumption:
11
Note the difference in contour Such a difference leads to variable selection
12
Parameter path
13
Computing the Lasso I: LARS
14
LARS
• Credit goes to Stanford statistics. • It is first designed for linear models. With nonlinear models, consider “transforming” into an iterative, weighted estimate
p
• In a similar spirit, we can smooth any two estimates we choose.
20
More serious problems with Lasso
• It is in general not selection consistent. More specifically, it over-selects. • The adaptive Lasso pen( β ) = λn ∑i =1 w j | β i |
• The straightforward approach: assume the logistic regression model and construct the likelihood function. • The robust rank based estimate
34
• There are a few other examples. • Common ground: the objective function H is a Ustatistic (as opposed to sum of iid terms). • Consider penalized selection
5
Linear model with orthogonal design
6
Why Ridge is not convenient
•All estimates are nonzero •Not all covariates are associated with the response •Possible external selection
9
Lasso
• A simple case: linear regression with orthogonal design. Here Lasso is equivalent to thresholding
10
Lasso
• A direct consequence: when the penalty is large enough, some estimates will be exactly zero • Only covariates with nonzero estimated regression coefficients enter the model. • That is, simultaneous estimation and variable selection. Note the difference from the significance level based.
pen ( β ) = λn ∑i =1 | β i | ;
p
γቤተ መጻሕፍቲ ባይዱ
γ = 1/ 2
24
• When γ → 0 , it goes to AIC/BIC type penalization • When γ → 1 , it goes to Lasso • Intuition: “more penalties” for smaller regression coefficients (the same as the adaptive Lasso)
22
Bridge
23
Bridge
• The adaptive Lasso demands an initial estimate. • With high dimensional data, the initial estimate may not exist. • A penalty that does not demand initial estimates: the bridge estimate
15
Computing the Lasso II: Boosting
16
Boosting
• Credit goes to Kim and Kim • This algorithm is applicable as long as the objective function is continuously differentiable • Problems: it is an approximation; it may take a while to converge. • There are other algorithms for computing Lasso.
31
32
Consider Linear Regression
• The likelihood based approach gives the ordinary least squares. • The robust rank based estimate:
33
Consider Binary Data under GLM
25
Bridge
26
Computational algorithm
27
Computational algorithm
28
Statistical properties
29
SCAD
• Developed by Prof. Jianqing Fan • It is estimation and selection consistent (one of the first “oracle” estimates) • Its definition looks “different” • I will not cover it. BUT, it does not mean it is not important.
Penalized Variable Selection
1
Modeling the Joint Effects
• Now consider the more familiar Y ~ f ( X 1 ,..., X p ) • For example, with binary Y, the logistic model with p covariates • Now, instead of multiple comparison, our biggest concern is over-fitting (saturated models)
4
Ridge penalization
• Consider additive, parametric covariate effects • Y ~ f ( β1 X 1 + ... + β p X p ) • pen ( β ) = λn ( β + ... + β )
2 1 2 p
• Key properties: unique estimates; all estimates are nonzero; Consistency with a fixed number of covariates; Computationally easy.
18
Some problems with Lasso
• When there are highly correlated covariates, it often selects only one (too sparse). • Solution: the elastic net p p pen ( β ) = λ1 ∑i =1 | β i | + λ2 ∑i =1 β i2 • It is a L1 penalty + a L2 penalty.