稀疏编码 Optimization with sparse inducingnorms

Yang Liu (ZJU) Sparse Optimization
if x ≥ λ |x| ≤ λ if x ≤ −λ
(10)
(11)
April 11, 2013 13 / 21
Coordinate Descent for the Lasso
The objective function:
1 min f (w) = (y − β 2
Optimization with Sparse Inducing Norms
Yang Liu
Zhejiang University Digital Computing & Design Lab liuy@
April 11, 2013
Yang Liu (ZJU)
Sparse Optimization
Yang Liu (ZJU) Sparse Optimization April 11, 2013 3 / 21
Improving Least Squares
Some improvements on Least Squares: Prediction accuracy: Shrinkage
Ridge Regression: minw Other loss function All Subsets Forward Selection
−
j =1
− 2 xij wj ))
(3)
The Lagrangian is:
n p p + xij wj − j =1 j =1 p − 2 xij wj )) + λ j =1 p + λ+ j wj j =1
Yang Liu (ZJU) Sparse Optimization
p − + ) ( wj + wj
Step 4: If a non-zero coefficient hits zero, drop its variable from the active set of variables and recompute the current joint least squares direction. It is also called homotopy algorithm.
April 11, 2013
1 / 21
Outline
1 2 3 4 5 6
Least Square Regression LASSO and LARS Bayes Point of View Coordinate Descend Algorithms Proximal Methods Conclusion
0
2
3 4
OMP solves a
regularized regression problem: minw y − Xw
2 2,
s.t.
w0≤λ源自(1)Yang Liu (ZJU)
Sparse Optimization
April 11, 2013
5 / 21
Least Angle Regression
Yang Liu (ZJU)
Sparse Optimization
April 11, 2013
6 / 21
LASSO
LASSO: Least Absolute Shrinkage and Selection Operator
minw y − Xw
2 2
+λ
w
1
(2)
Least Angle Regression: Lasso Modification
Algorithm 1 pseudocode for Least Angle Regressiont 1: Standardize the predictors to have mean zero and unit norm. Start with the residual r = y − y ¯, w1 , w2 , . . . , wp = 0. 2: Find the predictor xj most correlated with r . 3: Move wj from 0 towards its least-squares coefficient < xj , r >, until some other competitor xk has as much correlation with the current residual as does xj . 4: Move wj and wk in the direction defined by their joint least squares coefficient of the current residual on (xj , xk ), until some other competitor xl has as much correlation with the current residual. 5: Continue in this way until all p predictors have been entered. After min(N − 1, p) steps, we arrive at the full least-squares solution.
Yang Liu (ZJU) Sparse Optimization April 11, 2013 7 / 21
Sparse Coding and Dictionary Learning
The objective function
n
argmin :
{yi }n i=1 ,D i=1
xi − Dyi dj
Yang Liu (ZJU)
Sparse Optimization
April 11, 2013
11 / 21
Soft-thresholding operator
Let’s begin with the easiest problem: 1 minw (x − w)2 + λ|w|2 2 2
(7)
How about
Two reasons make us not satisfied with the least squares estimates: Prediction accuracy: the least squares estimates often have low bias but large variance. Interpretation: with a large number of predictors, we often would like to determine a smaller subset that exhibit the strongest effects.
KKT conditions for LASSO
Let w = w+ − w− , Lasso problem becomes:
n p p + xij wj j =1 p + − s.t. wj ≥ 0, wj ≥ 0, j =1 + − (wj + wj )≤s
minw+ ,w−
i=1
(yi − (
April 11, 2013
4 / 21
Orthogonal Matching Pursuit
Orthogonal Matching Pursuit: Forward stepwise selection
1
Initialize the residual r0 = y and initialize the set of selected variales X (c0 ) = ∅. Let t = 1. Find the variable Xt1 that solves the maximization problem: Tr maxt Xt and add the variable Xti to the set of selected i− 1 variables. Update ci = ci−1 ∪ {ti }. Let Pi = X (ci )(X (ci )T X (ci ))−1 X (ci )T If the stopping condition is achieved, stop the algorithm. Otherwise, set i = i + 1 and return to Step 2.
step-wise (MP, OMP) stage-wise
y − Xw
2 2
+λ
w
2 2
Interpretation: Subset Selection
Backward Selection
Least Angle Regression LASSO
Yang Liu (ZJU)
Sparse Optimization
2 Define f (w) = 1 2 (x − w ) + λ|w |1 , we have:
∂f (w) = x − w + λ sign(w) ∗ |z | ∂w where |z | ≤ 1. Set
∂f (w) ∂w
(9)
= 0, we have:
x − λ, ∗ w = 0, if x + λ, For similicity, Eq. (10) can also be written as wj = sign(x)(x − λ)+
xT j (y − Xw ) = λ sign(wj ) and |xT j (y − Xw )| ≤ λ
Yang Liu (ZJU) Sparse Optimization April 11, 2013
(5)
(6)
10 / 21
Bayes Inference