S. Kernel Basis Pursuit

Kernel Basis PursuitVincent Guigue,Alain Rakotomamonjy,St´e phane Canu1 Lab.Perception,Syst`e mes,Information-CNRS-FRE2645 Avenue de l’Universit´e,76801St´Etienne du Rouvray{Vincent.Guigue,Alain.Rakoto,Stephane.Canu}@insa-rouen.frR´e sum´e:Les m´e thodes`a noyaux sont largement utilis´e es dans le domaine de la r´e gression.Cependant,ce type de probl`e me aboutit`a deux questions r´e currentes: comment optimiser le noyau et comment r´e gler le compromis biais-variance? L’utilisation de noyaux multiples et le calcul du chemin complet de r´e gularisation permettent de faire face simplement et efficacement`a ces deux tˆa ches.L’intro-duction de noyaux multiples est´e galement un moyen de fusionner des sources d’information h´e t´e rog`e nes.Notre approche est inspir´e e de l’algorithme Basis Pursuit(Chen et al.,1998). Nous avons suivi la m´e thode de Vincent et Bengio pour la non-lin´e arisation du Basis Pursuit(Vincent&Bengio,2002).Cet article pr´e sente une solution simple et parcimonieuse pour le probl`e me de r´e gression par m´e thode`a noyaux multiples.Nous avons utilis´e la formulation du LASSO(Least Absolute Shrinkage and Selection Operator)(Tibshirani,1996), bas´e e sur une r´e gularisation L1,et l’algorithme du LARS(Stepwise Least Angle Regression)(Efron et al.,2004)pour la r´e r´e gularisation L1est un gage de parcimonie tandis que le calcul du chemin complet de r´e gularisation, via le LARS,permet de d´efinir de nouveaux crit`e res pour trouver le compro-mis biais-variance optimal.Nous pr´e senterons´e galement une heuristique pour le r´e glage des param`e tres du noyau,afin de rendre la m´e thode compl`e tement non param´e trique.Mots-cl´e s:R´e gression,Noyaux Multiples,LASSO,M´e thode Non-Param´e triqueAbstract:Kernel methods have been widely used in the context of regression. But every problem leads to two major tasks:optimizing the kernel and setting the fitness-regularization ing multiple kernels and Basis Pursuit is a way to face easily and efficiently these two tasks.On top of that,it enables us to deal with multiple and heterogeneous sources of information.Our approach is inspired by the Basis Pursuit algorithm(Chen et al.,1998).We use Vincent and Bengio’s method(Vincent&Bengio,2002)to kernelize the Basis Pursuit and introduce the ability of mixing heterogeneous sources of information.CAp2005This article aims at presenting an easy,efficient and sparse solution to the multipleKernel Basis Pursuit problem.We will use the Least Absolute Shrinkage andSelection Operator(LASSO)formulation(Tibshirani,1996)(L1regularization),and the Stepwise Least Angle Regression(LARS)algorithm(Efron et al.,2004)as solver.The LARS provides a fast and sparse solution to the LASSO.The factthat it computes the optimal regularization path enables us to propose new auto-adaptive hyper-parameters for thefitness-regularization compromise.We willalso propose some heuristics to choose the kernel parameters.Finally,we aim atproposing a parameter free,sparse and fast regression method.Key words:Regression,Multiple Kernels,LASSO,Parameter Free.1IntroductionThe context of our work is the following:we wish to estimate the functional dependency between an input x and an output y of a system given a set of examples{(x i,y i),x i∈X,y i∈Y,i=1...n}which have been drawn i.i.d from an unknown probability law P(X,Y).Thus,our aim is to recover the function f which minimizes the following riskR[f]=E{(f(X)−Y)2}(1) but as P(X,Y)is unknown,we have to look for the function f which minimizes theempirical risk:R emp[f]=ni=1(f(x i)−y i)2(2)This problem is ill-posed and a classical way to turn it into a well-posed one is to use regularization theory(Tikhonov&Ars´e nin,1977;Girosi et al.,1995).In this con-text,the solution of the problem is the function f∈H that minimizes the regularized empirical risk:R reg[f]=1Kernel Basis Pursuit method consisting in adding functions from a dictionary.The bias-variance problem involves several parameters,especially the kernel parameters and the hyper-parameter trading between goodness-of-fit and regularization.Our solution is based onℓ1regularization,we useΩ= β ℓ1in equation3.This for-mulation is called the Least Absolute Shrinkage and Selection Operator(LASSO)(Tib-shirani,1996),it will enable us to improve sparsity.Our solver relies on the Stepwise Least Angle Regression(LARS)algorithm(Efron et al.,2004),which is an iterative forward algorithm.Thus,the sparsity of the solution is closely linked to the efficiency of the method.We use Vincent and Bengio’s strategy(Vincent&Bengio,2002)to kernelize the resulting method.Finally,we end at the Kernel Basis Pursuit algorithm. Associated with this learning problem,there are two major tasks to build a good re-gression function with kernel:optimizing the kernel and choosing a good compromise betweenfitness and regularization.The use of multiple kernels is a way to make the first task easier.We will use the optimal path regularization properties of the LARS to propose new heuristics,in order to set dynamically thefitness-regularization compro-mise.In section2,we will compare two approaches to the question of sparsity:the Match-ing Pursuit and the Basis Pursuit.We will explain the building and the use of the multiple kernels,combined with the LARS in section3.Our results on synthetic and real data are presented in section4.Section5gives our conclusions and perspectives on this work.2Basis vs Matching PursuitTwo common strategies are available to face the problem of building a sparse regres-sion function f.Thefirst one relies on an iterative building of f.At each step k,the comparison between the target y and the function f k leads to add a new source of infor-mation to build f k+1.This approach is fast but it is greedy and thus sub-optimal.The second solution consists in solving a learning problem,by minimizing the regularized empirical risk of equation3.Mallat and Zhang introduced the Matching Pursuit algorithm(Mallat&Zhang,1993): they proposed to construct a regression function f as a linear combination of elemen-tary functions g picked from afinite redundant dictionary D.This algorithm is iterative and one new function g is introduced at each step,associated with a weightβ.At step k,we get the following approximation of f:f k=ki=1βi g i(5)Given R k,the residue generated by f k,the function g k+1and its associated weight βk+1are selected according to:(g k+1,αk+1)=argmin g∈D,β∈R R k−βg 2(6) The improvements described by Pati et al.(Orthogonal Matching Pursuit algorithm) (Pati et al.,1993)keep the same framework,but optimize all the weightsβi at eachCAp2005step.A third algorithm called pre-fitting(Vincent&Bengio,2002)enables us to choose (g k+1,βk+1)according to R k+1.All those methods are iterative and greedy.The different variations improve the weights or the choice of the function g but the main characteristic remains unchanged. Matchin Pursuit does not allow to get rid of a previous source of information,which means that its solution is sub-optimal.The approach of Chen et al.(Chen et al.,1998) is really different:they consider the whole dictionary of functions and look for the best linear solution(equation5)to estimate y,namely,the solution which minimizes theregularized empirical ingΩ= β ℓ1leads to the LASSO formulation.Such aformulation requires costly and complex linear programming(Chen,1995)or modified EM implementation(Grandvalet,1998)to be solved.Finally it enables them tofind an exact solution to the regularized learning problem.The Stepwise Least Angle Regression(LARS)(Efron et al.,2004)offers new oppor-tunities,by combining an iterative and efficient approach with the exact solution of the LASSO.The fact that the LARS begins with an empty set of variables,combined with the sparsity of the solution explains the efficiency of the method.The ability of deleting dynamically useless variables enables the method to converge to the exact solution of the LASSO problem.3Learning with multiple kernels3.1Building a multiple kernel regression functionVincent and Bengio(Vincent&Bengio,2002)propose to treat the kernel K exactly in the same way as the matrix X.Each column of K is then a source of information that can be added to the linear regression model f.Given an input vector x and a parametric mapping functionΦθdefined byΦθ:R d Fx→Φθ(x)=Kθ(x,·)(7)where F is the spanned feature space,we consider Kθ(x,.)as a source of information. It becomes easy to deal with multiple mapping functionsΦi.The multiple resulting kernels K i are placed side by side in a big matrix K:K= K1...K i...K N (8) N is the number of kernels.In this situation,each source of information K i(x j,·)is characterized by a point x j of the learning set and a kernel parameter i.The number of information sources is then s=nN and K∈R n×s.The learning problem becomes a variable selection problem where theβi coefficients can be seen as the weights of the sources of information.We simplify the notations:f=Ni=1n j=1βij K i(x j,·)=s i=1βi K(i,·)=Kβ(9)Kernel Basis PursuitIt is important to note that no assumption is made on the kernel K θwhich can be non-positive.K can associate kernels of the same type (e.g.Gaussian)with different parameter values as well as different types of kernels (e.g.Gaussian and polynomial).The resulting matrix K is neither positive definite or square.3.2LARSThe LARS (Efron et al.,2004)is a stepwise iterative algorithm which provides an exact to minimization of the regularized empirical risk (equation 3)with Ω= β ℓ1.We use the following formulation,which is equivalent to the LASSO:min β y −Kβ 2With respect to: β ℓ1≤t (10)We denote by βi the regression coefficient associated to the i th source of information and by ˆy (j )=Kβ(j )the regression function at step j .More generally,we will use exponent to characterize the RS is made of the following main steps:1.Initialization:the active set of information source A is empty,all βcoefficients are set to zero.putation of the correlation between the sources of information and the residue.The residue R is defined by R =y −ˆy .3.The most correlated source is added to the active set.A ={A arg max i,θ(|K T θ(x i ,·)R |)}(11)4.Definition of the best direction in the active set:−→u A This is the most expensivepart of the algorithm in time computation since it requires the inversion of thematrix K T A K A .5.The original part of the algorithm resides in the computation of the step γ.The idea is to compute γsuch as two functions are equi-correlated with the residue (cf Fig.1)whereas Ordinary Least Square (OLS)algorithm defines γsuch as −→u A and −−−−−→ˆy (j +1),y become orthogonal.6.The regression function is updated:ˆy (j +1)=ˆy (j )+γ−→u A (12)It is necessary to introduce the ability of suppressing a function from the active set to fit the LASSO solution,namely to turn the forward algorithm into a stepwise method.When the sign of a βi changes during the update (equation (12),the step γis reduced so that this βi becomes zero.Then,the corresponding source is removed from the active set and an optimization is performed over the new active set.Solving the LASSO is really fast with this method,due to the fact that it is both forward and sparse.The first steps are not expensive,because of the small size of theCAp2005active set,then it becomes more and more time consuming with iterations.But the sparsity ofℓ1regularization limits the number of required RS begins with an empty active set whereas linear programming and other backward methods begin with all functions and require to solve high dimensional linear system to put irrelevant coefficients to zero.Given the fact that only one point is added(or removed)during an iteration,it is possible to update the inverted matrix of step four instead of fully computing it.This leads to a simple-LARS algorithm,similarly to the simple-SVM formulation(Loosli et al.,2004),which also increases the speed of the method.3.3Optimization of regularization parameterOne of the most interesting property of the LARS is the fact that it computes the whole regularization path.The regularization parameterλof equation3is equivalent to the bound t of equation10.At each step,the introduction of a new source of information leads to an optimal solution,corresponding to a given value of t.In the other classical algorithms,λis set a priori and optimized by cross-validation.The LARS enables us to compute a set of optimal solutions corresponding to different values of t,with only one learning stage.It also enables us to optimize the value of t dynamically,during the learning stage.Finding a good setting for t is very important:when t becomes too large,the re-sulting regression function is the same as the Ordinary Least Square(OLS)regression function.Hence,it requires the resolution of linear system of size s×s.Early stopping should enable us to decrease the time computation(which is linked to the sparsity of the solution)as well as to improve the generalization of the learning(by regularizing).3.3.1Different compromise parametersThe computation of the complete regularization path offers the opportunity to set the compromise parameter dynamically(Bach et al.,2004).Thefirst step is to look for different expressions of the regularization parameter t of equation(10).The aim is to find the most meaningful one,namely the easiest way to set this parameter.Kernel Basis Pursuit -The original formulation of the LARS relies on the compromise parameter t which is a bound on the sum of the absolute values of theβcoefficients.t is difficult to set because it is somewhat meaningless.-It is possible to apply Ljung criterion(Ljung,1987)on the autocorrelation of the residue.The parameter is then a threshold which decides when the residue can be considered as white noise.-Another solution consists in the study of the evolution of loss functionℓ(y i,f jθ(x i))with regards to the step j.The criterion is a bound on the variation of this cost.-ν-LARS.It is possible to define a criterion on the number of support vectors or on the rate of support vectors among the learning set.It is important to note that theνthreshold is then a hard threshold,whereas in theν-SVM method whereνcan be seen as an upper bound on the rate of support vectors(Sch¨o lkopf&Smola, 2002).However,all these methods require the setting of a parameter a priori.The value of this parameter is estimated by cross-validation.3.3.2Trap sourceWe propose another method based on a trap parameter.The idea is to introduce one or many sources of information that we do not want to use.When the most correlated source with the residue belongs to the trap set,the learning procedure is stopped.The trap sources of information can be built on different heuristics:-according to the original signal noise when there exists prior knowledge on the data,-with regards to the distribution of the learning points,to prevent overfitting(cfFig.2),in this case,a Gaussian kernel K=Kσof is added to the informationsources,withσof very small,-by adding random variables among the sources of information(with Gaussian or uniform distribution).This kind of heuristic has already been used in variable selection(Bi et al.,2003).The use of a trap scale is closely linked to the way that LARS selects the sources of information.As seen in section3.2,the selected source of information at a given iteration is the most correlated with the residue.Those heuristics are based on the meaning of the trap scale:the learning stage should be stopped when the residue is most correlated respectively with the noise,with only one source of information or with an independent random variable generated according to the uniform distribution. This means that no more relevant information is present in the sources that are not in the active set.CAp2005Figure2:Illustration of a trap scale based on overfitting heuristic(Gaussian kernel).When Kσ2(x,·)is the most correlated source of information with the residue,it meansthat the error is caused by only one point,it is a way to detect the beginning of overfit-ting.3.4Optimizing kernel parametersInstead of searching an optimal parameter(or parameter vector)for the problem,we propose tofind a key scale tofit the regions where there are the highest point density in the input space.We aim atfinding a reference Gaussian parameter so that the two near-est points in the input space have few influence on each other.This reference parameter represents the smallest bandwidth which can be interesting for a given problem.Then, we propose to build a series of bigger Gaussian parameters from this scale tofit the different densities of points that can append in the whole input space.A one nearest neighbors is performed on the training data.To describe high density regions,we focus on the shortest distances between neighbors.The key distance D k is defined as the distance between x i and x j,the two nearest points in the input space.The corresponding key Gaussian parameterσk is defined so that:K(x i,x j)=12πσk exp −D2kKernel Basis Pursuit4ExperimentsWe illustrate the efficiency of the methodology on synthetic and real data.Tables1and 2present the results with two different algorithms:the SVM and the LARS.We use four strategies to stop the learning stage of the LARS.-LARS- i|βi|is the classical method where a bound is defined on the sum of the regression coefficient.This bound is estimated by cross validation.-ν-LARS is based on the fraction of support vectors.νis also estimated by cross validation.-LARS-RV relies on the introduction of random variables as sources of informa-tion.The learning stage is stopped when one of these sources is picked up as most correlated with the residue.-LARS-σs relies also on a trap scale,but this scale is built according to the dis-tribution the learning set.Selecting a source in this trap scale can be seen as overfitting.We useσs=σk of equation(13).To validate this approach,we compare the results with classical Gaussianǫ-SVM re-gression,Parametersǫ,C andσare optimized by cross validation.In order to distin-guish the benefit of the LARS from the benefits of the multiple kernel learning,we also give the results of LARS algorithm combined with single kernel.4.1Synthetic dataThe learning of cos(exp(ωt))regression function,with random sampling show the mul-tiple kernel interest.We try to identify:f(t)=cos(exp(ωt))+b(t)(15) where b(t)is a Gaussian white noise of varianceσ2b=0.4.t∈[0,2]is drawn ac-cording to a uniform distribution,ω=2.4.We also tested the method over classical synthetic data described by Donoho and Johnstone(Donoho&Johnstone,1994).For those signals,we took t∈[0,1],drawn according to a uniform distribution.We use200points for the learning set and1000points for the testing set.The noise is added only on the learning set.Parameters(ν, i|βi|...)are computed by cross validation on the learning set.Table1presents the results over30runs for each data base.These results point out the sparsity and the efficiency of LARS solutions.Figure3 illustrates how multiple kernel learning enables the regression function tofit the local frequency of the model.It also shows that selected points belong higher and higher scales with iterations.Indeed,the correlation with the residue can be seen as an ener-getic criterion:when the amplitude of the signal remain constant,there is more energy in the low frequency part of the signal.That is why thefirst selected sources of infor-mation describe those parts of the signal.The results with different Donoho’s synthetic signals enable us to distinguish the benefits of the LARS method from the benefits ofCAp 2005Algorithmǫ-SVM ν-LARS cos(exp(t ))0.16±0.016155.40.17±0.01512000.041±0.006837.1000.039±0.005932.800Blocks 1.18±0.2045.3001.19±0.17210.028±0.007025.6340.028±0.006533.933HeaviSine 0.48±0.1251.43110.49±0.12404Algorithm LARS-P i |βi |LARS-RV cos(exp(t ))0.13±0.014122.4170.13±0.016121.470.035±0.00624600.035±0.007549.800Blocks 0.95±0.3434.02130.96±0.2542.3560.032±0.00502700.033±0.005927.330HeaviSine 0.50±0.1451.3000.49±0.1548.224Table 1:Results of SVM and LARS for the cos(exp(t ))and Donoho’s classical func-tions estimation.Mean and standard deviation of MSE on the test set (30runs),number of support vectors used for each solution,number of best performances.Kernel Basis PursuitFigure3:These illustratingfigures explain the learning of the cos(exp(ωt))function with low noise level.Selected learning points belong to different scales,depending on the local frequency of the function to learn.Rightfigure shows that selected points belong higher and higher scales with iterations,namely,the sources of information correlated with the residue are more and more local with iterations.the multiple kernels.The LARS improves the sparsity of the solution,whereas the multiple kernels improve the results on signals that require a multiple scale approach.ǫ-SVM achieves the best results for Ramp and HeaviSine signals.This can be ex-plained by the fact that the Ramp and HeaviSine signals are almost uniform in term of frequency.Theǫtube algorithm of the SVM regression is especially efficient on this kind of problem.It is important to note that LARS-RV and LARS-σs are parameter free methods when combined with the heuristic described in section3.4.Best results are achieved with LARS- i|βi|,however,LARS-RV results are almost equivalent without any parame-ters.4.2Real dataExperiments are carried out over regression data bases pyrim and triazines available in the UCI repository(Blake&Merz,1998).We compare our results with(Chang&Lin, 2005).The experimental procedure for real data is the following one:Thirty training/testing set are produced randomly,table2presents mean and standard deviation of MSE(mean square error)on the test set.80%of the points are used for training and the remaining paramters(ν, i|βi|...)are computed by cross validation on the learning set.The results obtained with LARS algorithm are either equivalent to Chang and Lin’s ones or better.ǫ-SVM solution is not really competitive but it gives an interesting information on the number of support vectors required for each RS-RV and LARS-σs results are very interesting:they are parameter free using the heuristic describe in section3.4,moreover the LARS-RV achieves the best results for pyrim.CAp2005Algo SVMǫ-SVM RV0.009±0.01637.1100.011±0.00931.20triazines0.021±0.005−−0.022±0.00637.00Algo LARSPi|βi|σs0.007±0.00638.08170.020±0.00552.038Table2:Results of SVM and LARS for the different regression database.Mean and standard deviation of MSE on the test set(30runs),number of support vectors used for each solution,number of best performances.5ConclusionThis paper enables us to meet two objectives:proposing a sparse kernel-based solution for the regression problem and introducing new solutions for the bias-variance compro-mise problem.The LARS offers opportunities for both problems.It gives an exact solution to the LASSO problem,which is sparse due toℓ1regularization.The ability of dealing with multiple kernels allows rough setting for the kernel parameters.Then,LARS algo-rithm optimizes the parameters at each iteration,selecting a new point in the optimal scale.The fact that the LARS computes the regularization path offers efficient and non parametric settings for the compromise parameter.This methodology gives good results on synthetic and real data.In the meantime,the required time computation is reduced compared with SVM,due to the sparsity of the obtained solutions.The perspectives of this work are threefold.We have to test LARS-methods on more databases to evaluate all properties.We also want to improve the multiple kernel build-ing.Indeed,the use of the currentσk often leads to a slight overfitting and to less sparse solutions.Finally,we will analyze the LARS-RV results deeper,to explain the good results and possibly improve them.ReferencesB ACH F.,T HIBAUX R.&J ORDAN M.(2004).Computing regularization paths for learningKernel Basis Pursuitmultiple kernels.In Advances in Neural Information Processing Systems,volume17.B I J.,B ENNETT K.,E MBRECHTS M.,B RENEMAN C.&S ONG M.(2003).Dimensionality reduction via sparse support vector machines.Journal of Machine Learning Research,3,1229–1243.B LAKE C.&M ERZ C.(1998).UCI rep.of machine learning databases.C HANG M.&L IN C.(2005).Leave-one-out bounds for support vector regression model se-lection.Neural Computation.C HEN S.(1995).Basis Pursuit.PhD thesis,Department of Statistics,Stanford University.C HEN S.,D ONOHO D.&S AUNDERS M.(1998).Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing,20(1),33–61.D ONOHO D.&J OHNSTONE I.(1994).Ideal spatial adaptation by wavelet shrinkage. Biometrika,81,425–455.E FRON B.,H ASTIE T.,J OHNSTONE I.&T IBSHIRANI R.(2004).Least angle regression. Annals of statistics,32(2),407–499.G IROSI F.,J ONES M.&P OGGIO T.(1995).Regularization theory and neural networks archi-tectures.Neural Computation,7(2),219–269.G RANDVALET Y.(1998).Least absolute shrinkage is equivalent to quadratic penalization.In ICANN,p.201–206.K IMELDORF G.&W AHBA G.(1971).Some results on Tchebycheffian spline functions.J. Math.Anal.Applic.,33,82–95.L JUNG L.(1987).System Identification-Theory for the User.L OOSLI G.,C ANU S.,V ISHWANATHAN S.,S MOLA A.J.&C HATTOPADHYAY M.(2004). Une boˆıte`a outils rapide et simple pour les svm.In CAp.M ALLAT S.&Z HANG Z.(1993).Matching pursuits with time-frequency dictionaries.IEEE Transactions on Signal Processing,41(12),3397–3415.P ATI Y.C.,R EZAIIFAR R.&K RISHNAPRASAD P.S.(1993).Orthogonal matching pursuits: recursive function approximation with applications to wavelet decomposition.In Proceedings of the27th Asilomar Conference in Signals,Systems,and Computers.S CH¨OLKOPF B.&S MOLA A.(2002).Learning with kernels.T IBSHIRANI R.(1996).Regression shrinkage and selection via the lasso.J.Royal.Statist., 58(1),267–288.T IKHONOV A.&A RS´E NIN V.(1977).Solutions of ill-posed problems.W.H.Winston.V INCENT P.&B ENGIO Y.(2002).Kernel matching pursuit.Machine Learning Journal,48(1), 165–187.W AHBA G.(1990).Spline Models for Observational Data.Series in Applied Mathematics, V ol.59,SIAM.。