sparsity models (tong)

Modern datasets are often high dimensional
statistical estimation suffers from curse of dimensionality
Sparsity: popular assumption to address curse of dimensionality
T. Zhang (Rutgers)
Sparsity Models
7 / 28
KKT Condition for Lasso Solution
Lasso solution: ˆL1 = arg min β
β
Y − Xβ
2 2
+λ β
1
ˆ=β ˆL1 : KKT condition: at β Exists a sub-gradient being zero: for all j = 1, . . . , p (Xj is the j -th column of X ): ˆ − y ) + λ∇|β ˆj | = 0. 2Xj (X β 1 Subgradient of L1 norm: ∇|u | = sign(u ) = −1 ∈ [−1, 1] u>0 u<0 u = 0.
Sparsity Models
4 / 28
Standard Sparse Regression
¯+ Model: Y = X β
Y ∈ R n : observation X ∈ R n×p : design matrix ¯ ∈ R p : parameter vector to be estimated β ∈ R n : zero mean stochastic noise with variance σ 2
Are efficient algorithms (such as L1 or OMP) good enough?
T. Zhang (Rutgers)
Sparsity Models
7 / 28
Conditions and Results
Type of results (sparse recovery):
β 2 2
supp(β ) ⊂ F
s.t. |F | ≤ k
Forward Greedy Algorithm (OMP): select variables one by one
Initialize variable set F k = ∅ at k = 0 Iterate k = 1, . . . , p
Some extensions (complex regularization)
structured sparsity graphical model matrix regularization
T. Zhang (Rutgers)
Sparsity Models
2 / 28
Modern Sparsity Analysis: Motivation
β 2 2
supp(β ) ⊂ F
s.t. |F | ≤ k
Forward Greedy Algorithm (OMP): select variables one by one
Initialize variable set F k = ∅ at k = 0 Iterate k = 1, . . . , p
Variable selection (can we find nonzero variables): can we recover the ¯? true support F ¯? ˆ) ≈ F supp(β ¯): can we recover Parameter estimation (how well we can estimate β the parameters? ˆ−β ¯ 2 β 2 ≤?
T. Zhang (Rutgers)
Sparsity Models
6 / 28
Greedy Algorithms for standard sparse regularization
Reformulation: find variable set F ⊂ {1, . . . , p} to minimize min X β − Y
Are efficient algorithms (such as L1 or OMP) good enough? Yes but require conditions:
Irrepresentable: for support recovery RIP – Restricted Isometry Property: for parameter recovery
5 / 28
Algorithms for Standard Sparsity
L0 regularization: natural method (computationally inefficient) ˆL = arg min Y − X β β 0
β 2 2,
subject to β 0 ≤ k
Y ∈ R n : observation X ∈ R n×p : design matrix ¯ ∈ R p : parameter vector to be estimated β ∈ R n : zero mean stochastic noise with variance σ 2
T. Zhang (Rutgers)
High dimensional setting: n p ¯ Sparsity: β has few nonzero components
¯) = {j : β ¯j = 0}. supp(β ¯ 0 = |supp(β ¯)| is small: β n
T. Zhang (Rutgers)
Sparsity Models
High dimensional setting: n
p
T. Zhang (Rutgers)
Sparsity Models
4 / 28
Standard Sparse Regression
¯+ Model: Y = X β
Y ∈ R n : observation X ∈ R n×p : design matrix ¯ ∈ R p : parameter vector to be estimated β ∈ R n : zero mean stochastic noise with variance σ 2
Theoretical question: recovery performance?
T. Zhang (Rutgers)
Sparsity Models
6 / 28
Conditions and Results
Type of results (sparse recovery):
Variable selection (can we find nonzero variables): can we recover the ¯? true support F ¯? ˆ) ≈ F supp(β ¯): can we recover Parameter estimation (how well we can estimate β the parameters? ˆ−β ¯ 2 β 2 ≤?
Modern datasets are often high dimensional
statistical estimation suffers from curse of dimensionality
T. Zhang (Rutgers)
Sparsity MoAnalysis: Motivation
find best variable j to add to F k −1 (maximum reduction of squared error) F k = F k −1 ∪ {j }
terminate with some criterion; ˆ using regression with selected variables F k output β
4 / 28
Algorithms for Standard Sparsity
L0 regularization: natural method (computationally inefficient) ˆL = arg min Y − X β β 0
β 2 2,
subject to β 0 ≤ k
find best variable j to add to F k −1 (maximum reduction of squared error) F k = F k −1 ∪ {j }
terminate with some criterion; ˆ using regression with selected variables F k output β
T. Zhang (Rutgers)
Sparsity Models
5 / 28
Greedy Algorithms for standard sparse regularization
Reformulation: find variable set F ⊂ {1, . . . , p} to minimize min X β − Y
L1 regularization (Lasso): convex relaxation (computationally efficient) ˆL = arg min β 1
β
Y − Xβ
2 2
+λ β
1
Theoretical questions:
¯ (recovery performance) how well can we estimate parameter β