Mutual information in learning feature transformations

Kari Torkkola A540AA@ William M.Campbell P27439@ Motorola Human Interface Lab,2100East Elliot Road,MD EL508,Tempe,AZ85284,USAAbstractWe present feature transformations useful for ex-ploratory data analysis or for pattern recognition.Transformations are learned from example datasets by maximizing the mutual information be-tween transformed data and their class labels.Wemake use of Renyi’s quadratic entropy,and weextend the work of Principe et al.to mutual in-formation between continuous multidimensionalvariables and discrete-valued class labels.1.IntroductionReducing the dimensionality of feature vectors is usually an essential step in pattern recognition tasks to achieve practical ually this is done by using domainknowledge or heuristics.Dimensionality reduction is also essential in exploratory data analysis,where the purpose often is to map data onto a low-dimensional space for hu-man eyes to gain some insight to the data.It is well known that principal component analysis(PCA)has nothing to do with discriminative features optimal for classification,since it is only concerned with the covari-ance of all data regardless of the class.However,it may bevery useful in reducing noise in the data.So called linear discriminant analysis(LDA)can be used to derive a dis-criminative transformation that is optimal for certain cases(Fukunaga,1990).LDAfinds eigenvectors ofwhere is the between-class covariance matrix,andis the sum of within-class covariance matrices.The eigen-vectors corresponding to largest eigenvalues of form the columns of the transformation matrix,and new discrim-inative features are derived from the original ones sim-ply by.This simple algebraic way of deriving the transformation matrix is both a strength and a weaknessof the method.Since LDA makes use of only second-order statistical information,covariances,it is optimal for data where each class has a unimodal Gaussian density with well separated means.Also,the maximum rank of is ,where is the number of different classes.Thus LDA cannot produce more than features.Although extensions have been proposed for the latter problem(e.g., see Okada&Tomita,1985),thefirst one remains.The purpose of this paper is to demonstrate that mutual information(MI)can act as a more general criterion that overcomes these limitations.It accounts for higher-order statistics,not just for second order,and it can also be used as the basis for non-linear transformations.A result by Fano(1961)exists which states that maximizing the mu-tual information between transformed data and their class labels achieves the lowest possible bound to the error of a classifier.We discuss this result in Section3.The reasons why mutual information is not in wider use currently lie in computational difficulties.The probabil-ity density functions of the variables are required,and MI involves integrating functions of those,which in turn in-volves evaluating them on a dense set,which leads to ex-ponential computational complexity.Evaluating MI between two scalar variables is feasible through histograms,and this approach has found use in feature selection rather than in feature transformation(Bat-titi,1994;Bonnlander&Weigend,1994;Yang&Moody, 1999).However,greedy algorithms based on sequential feature selection using MI are suboptimal,as they fail to find a feature set that would jointly maximize the MI be-tween the features and the labels.This failure is due to the sparsity of(any amount of)data in high-dimensional spaces for histogram-based MI estimation.Feature selec-tion through any other joint criteria also leads to a combi-natorial explosion.In fact,for this very reasonfinding a transformation to lower dimensions might be easier than selecting features, given an appropriate objective function.Principe,Fisher,and Xu(2000)have shown that using an unconvential definition of entropy,Renyi’s entropy instead of Shannon’s,can lead to expressions of mutual informa-tion with significant computational savings.They explored the mutual information between two continuous variables. We describefirst the basis of their work,and then extend it to mutual information between continuous variables and discrete class labels.We use the criterion to learn linear dimension-reducing feature transformations with optimal discriminative ability,and we demonstrate the results with some well-known data sets both in visualization and pattern recognition applications.2.Formalized ObjectiveGiven a set of training data,as samples of a continuous-valued random variable,,and class labels as samples of a discrete-valued random vari-able,,the objective is to find a transformation(or its parameters) that maximizes the mutual information(MMI)be-tween transformed data and class labels.To this end we need to express as a function of the data set,,in a differentiable form.Once that is done, we can perform gradient ascent on as follows(4)after applying the identities and.Mutual information also measures independence between two variables,in this case between and.It equals zero when,that is,when the joint density of and factors(the condition for independence).How-ever,practical evaluation of MI based on this expression is difficult.3.2Fano’s BoundWhat does mutual information have to do with optimal classification?The answer is given by Fano’s(1961)in-equality.This result determines a lower bound to the prob-ability of error when estimating a discrete random variable from another random variable as(5) where is the estimate of after observing a sample of, which can be scalar or multivariate.Thus the lower bound on error probability is minimized when the mutual infor-mation between and is maximized,or,finding such features achieves the lowest possible bound to the error of a classifier.Whether this bound can be reached or not,de-pends on the goodness of the classifier.4.Renyi’s Entropy Reduces to Pairwise InteractionsHaving established that mutual information is an optimal measure to learn feature transformations,we recapitulate the results of Principe,Fisher,and Xu(2000)of using Renyi’s entropy combined with Parzen window density es-timation to reduce computational complexity.4.1Renyi’s EntropyInstead of Shannon’s entropy we apply Renyi’s quadratic entropy as described in(Principe et al.,2000)because of its computational advantages.For a discrete variable, and for a continuous variable Renyi’s quadratic entropy is defined as(6)(7)It has been shown that for the purposes offinding the ex-trema,coincides with Shannon’s entropy(Kapur, 1994).Later in Section5following Principe et al.we de-velop expressions for quadratic mutual information based on Renyi’s entropy measure.4.2Parzen Density EstimationWhether based on Shannon’s or Renyi’s entropy,evaluating the MI requires estimates of the probability density func-tions of the variables involved.One nonparametric method for this estimation is the Parzen(1962)window method.This involves placing a kernel function on top of each sam-ple and evaluating the density as the sum of the kernels. It turns out that Renyi’s measure combined with Parzen density estimation method using Gaussian kernels provides significant computational savings.The Gaussian kernel in-dimensional space is defined(8) Now,for two kernels,the following holds(9) Thus the convolution of two Gaussians centered at and is a Gaussian centered at with covariance as a sum of the original covariances.This property facilitates evaluating Renyi’s quadratic entropy measure which is a function of the square of the density function.Assume that the density is estimated as a sum of symmetric Gaussians each centered at a sample as(12) This can also be understood as the Kullback-Leibler di-vergence between the joint density and the product of the marginal densities of the variables.Thus(13) In general,for two densities and the divergence is written as(17) they writeIt is easy to see that both measures are always positive,and when both measures evaluate to zero.Thisjustifies their use in minimizing the mutual information. Moreover,especially is also suitable for maxi-mizing the mutual information(Principe,personal commu-nication,1999),although a rigorous proof for this does not exist.Since mutual information is expressed as the divergence between the joint density and the marginals,we can insert them into the two different quadratic divergence expres-sions.Cauchy-Schwartz inequality results in the following quadratic mutual information measure between two contin-uous variables and:(23)(24) Let us make the following definitions of the quantities ap-pearing in(23)and(24)(25) With these,that is needed in(1)will be(31) Since the joint density can be written aswe havefor each.The density of all data is,thus we have(33)Now we write the quantities in(25)using a set of samplesin the transformed space by inserting(30),(32),and(33)into(25).Making use of(9)and(11)we get:(39)This represents a sum of forces that other“particles”re-gardless of class exert to particle.Note that this par-ticle is also denoted as when the class is irrelevant.Direction is again towards.The derivative of is7.2Parametrization as RotationsTo be able to use optimization algorithms that work best in unconstrained problems,we parametrize the linear trans-formation in terms of rotations.Let us begin with an or-thonormal basis,for example,byfilling the columns of with the thefirst unit vectors.Now,there arefree parameters(rotation angles in this case).This number can be derived by considering the basis vectors.A rota-tion plane can be defined by its normal vector around which the current basis vector set is rotated by angle,or alternatively,by any two vectors that span.It does not make sense to rotate the basis vectors along a plane that is defined by two of the basis vectors themselves,since the projection of the data onto the subspace would remain un-changed after the rotation.However,combining each of the current basis vectors with an orthogonal basis vector of the remaining space defines planes.Rotation of along any of these planes will now change the projec-tion.Thus we have independent rotation angles as parameters(instead of coefficients in).These are easily represented as Givens rotation matrices.Let us define as a unit matrix with the exeption of four elements,,,,and.A rotation plane is now defined by dimensions and,and rotates the columns of by angle along that plane.Thus we can write(42)where definingchanges the indexing into a more convenient form.Required derivatives can be computed as follows.This matrix is other-wise a zero matrix,except the trigonometric functions in have been replaced by their derivatives.A gradient ascent algorithm for updatingis now(44)8.ExperimentsWe present experiments in two types of applications,data visualization and pattern recognition,with data sets that are available on the Internet.In the former application,a pro-jection is learned from a-dimensional feature space onto a plane for visualization purposes.In the latter,wefind a projection onto a range of dimensions between one and,and we compare the goodness of the MMI projec-tion to PCA and LDA projections by using two different pattern recognition methods,Learning Vector Quantization (LVQ)(Kohonen et al.,1992),and a polynomial classifier (Campbell&Assaleh,1999).8.1Visualization of Class Structure in DataThis example illustrates a projection from36-dimensional feature space onto two.1This is the Landsat satellite image database from UCI Machine Learning Repository.2The data has six classes,and we used1500of the training sam-ples.The LDA projection is presented in Figure1,and the MMI-projection in Figure2.LDA separates two of the classes very well but places the other four almost on top of each other.The criterion of LDA is a combination of representing each class as compactly as possible and as separated from each other as possible.The compromise in Figure1has achieved this:all classes are represented as quite compact clusters–unfortunately they are on top of each other.Two of the classes and a third cluster compris-ing of the four remaining classes are well separated.MMI has produced a projection that attempts to separate all of the classes.We can see that the four classes that LDA was not able to separate lie on a single continuum and blend into each other,and MMI has found a projection orthogonal to that continuum while still keeping the other two classes well separated.In this example,as well as in general,it is essential to use a kernel width that gives each sample an influence over every other sample.A simple rule that seems to work well is to take the distance of two farthest points in the output space, and use a kernel width of about half of that.The forces become more local if the kernel width is narrowed down.If the transformation has more degrees of freedom,this sug-gests a procedure where a narrow kernel could be used at the end tofine-tune the parameters of the transformation.8.2Classification ExperimentsClassification experiments were performed usingfive data sets very different in terms of dimensionality,number of classes,and the amount of data.The sets and some of their characteristics are presented in Table1.The Phoneme set is available with the LVQ1More examples and video clips illustrating convergence are available at /torkkola/mmi.html 2/mlearn/MLRepository.html3http://www.cis.hut.fi/research/software.shtml−3−2−1123−3−2−1123Figure 1.The Landsat image data set.LDA projection.Figure 2.The Landsat image data set.MMI projection.is available from the Aston University.4The rest of the data sets are from the UCI Machine Learning plete training sets were used to learn the MMI projec-tion with two exceptions.Some 1500random samples were used with the Landsat set,and 2000samples with the Let-ter set.This was done because of the computational com-plexity of of the training procedure,where is the number of samples.This is due to evaluation of pairwise distances in the output space.PCA and LDA projections were learned from the complete training sets each case.For each data set and each dimension the MMI projection was initialized with PCA,LDA (where possible),and three different random projections.After Levenberg-Marquardt optimization the projection with the highest mutual infor-mation was chosen as the final projection for the particular dimension.The learned transformation was then applied to the complete training and test sets to produce the training and test sets for the two classifiers.Data set Dim.Classes Train size Test sizeLVQ experiments were performed using packageLVQTable2.Accuracy on the Letter data set using LVQ classifier.Dim:PCA13.438.053.168.180.386.392.4MMI12349153641.281.585.887.889.490.390.4LDA65.182.086.486.287.689.590.4 Table4.Accuracy on the Phoneme data set using LVQ classifier.Dim:PCA5.166.074.780.282.886.090.0MMI1234571241.588.087.889.796.497.299.0LDA99.499.198.999.298.999.099.0 Table6.Accuracy on the Pima data set using LVQ classifier.Dim:PCA65.8------MMI123456856.377.375.874.677.777.778.1LDA80.580.979.380.579.379.378.1 Although we only describe linear transformations in this paper,nothing prevents applying this method to non-linear transformations,as long as they are parametrizable and dif-ferentiable.In fact,the potential of the method lies exactly here,since unlike LDA,the MMI criterion is readily appli-cable to non-linear cases.ReferencesBattiti,R.(1994).Using mutual information for selecting features in supervised neural net learning.Neural Net-works,5,537–550.Bonnlander,B.,&Weigend,A.S.(1994).Selecting in-put variables using mutual information and nonparamet-ric density estimation.Proceedings of the1994Inter-national Symposium on Artificial Neural Networks(pp. 42–50).Tainan,Taiwan.Campbell,W.M.,&Assaleh,K.T.(1999).Polynomial classifier techniques for speaker verification.Proceed-ings of the IEEE International Conference on Acoustics, Speech,and Signal Processing(pp.321–324).Phoenix, AZ,USA.Fano,R.(1961).Transmission of information:A statistical theory of communications.New York:Wiley. Fukunaga,K.(1990).Introduction to statistical pattern recognition(2nd edition).New York:Academic Press. Kapur,J.(1994).Measures of information and their appli-cations.New Delhi,India:Wiley.Kohonen,T.,Kangas,J.,Laaksonen,J.,&Torkkola,K. (1992).LVQ。