Distinctiveness-sensitive nearest-neighbor search for efficient similarity retrieval of mul

Distinctiveness-Sensitive Nearest-Neighbor Search for Efficient Similarity Retrieval of Multimedia InformationNorio Katayama and Shin’ichi SatohNational Institute of Informatics2-1-2Hitotsubashi,Chiyoda-ku,Tokyo101-8430,JapanE-mail:katayama,satoh@nii.ac.jpAbstractNearest neighbor(NN)search in high dimensional fea-ture space is widely used for similarity retrieval of multi-media information.However,recent research results in the database literature reveal that a curious problem happens in high dimensional space.Since high dimensional space has high degree of freedom,points could be so scattered that every distance between them might yield no significant difference.In this case,we can say that the NN is indistinc-tive because many points exist at the similar distance.To make matters worse,indistinctive NNs require more search cost because search completes only after choosing the NN from plenty of strong candidates.In order to circumvent the harmful effect of indistinctive NNs,this paper presents a new NN search algorithm which determines the distinc-tiveness of the NN during search operation.This enables us not only to cut down search cost but also to distinguish distinctive NNs from indistinctive ones.These advantages are especially beneficial to interactive retrieval systems. 1.IntroductionNearest-neighbor(NN)search in the feature space is widely used for the similarity retrieval of multimedia infor-mation.Each piece of multimedia information is mapped to a vector in a multi-dimensional space where the distance between two vectors(typically,Euclidean distance between the heads of vectors)corresponds to the similarity of multi-media information.These vectors and the space are called feature vectors and feature space respectively.Once the fea-ture space is obtained,the similarity retrieval of multimedia information is reduced to NN search in the feature space. One of the important properties of the feature space is high dimensionality.It is very common to use10or higher di-mensional space for the feature space.Recent results in the literature reveal that a curious prob-lem happens in high dimensional space,which is not imag-ined in low dimensional space.Since high dimensional space has high degree of freedom,points could be so scat-tered that every distance between them might yield no sig-nificant difference.A typical example appears in the uni-formly distributed points,i.e.,points distributed uniformly in a unit hypercube.For example,Berchtold et al.[1]re-ported that most of the points are located near the surface of the cube,and Katayama et al.[2]reported experimen-tally that the distances among the data points are very sim-ilar for any combination of them.Recently,Beyer et al.[3] showed that under a broad set of conditions(broader than uniform distribution)the distance to the nearest data point approaches the distance to the farthest data point as dimen-sionality increases.Thus,in high dimensional space,the distance to the nearest neighbor may not yield significant difference compared with the distances to the other points. In this case,we can say that the nearest neighbor is indis-tinctive because many points exist at the similar distance with the nearest neighbor.To make matters worse,indis-tinctive nearest neighbors require more search cost than dis-tinctive ones because search completes only after choosing the nearest neighbor from plenty of strong candidates.Indistinctive nearest neighbors are troublesome with re-spect to the similarity retrieval.They are less informative to users since we do not have significant difference between the nearest neighbor and the others.In addition,indistinc-tive nearest neighbors degrade the system response because they incur high search cost.As mentioned above,high di-mensional space is more likely to have indistinctive nearest neighbors.This means that the similarity retrieval based on high dimensional feature space is more vulnerable to the harmful effect of indistinctive nearest neighbors.However, this does not mean that high dimensional feature space is useless for the similarity retrieval of multimedia informa-tion.Since the point distribution of the feature space is skewed,the intrinsic dimensionality(effective dimensional-ity)in a local region can be much less than the dimensional-ity of the feature space.Hence,we might have indistinctive nearest neighbors in one region but might have distinctive ones in another.The distinctiveness of the nearest neighbor differs from one local region to another.D i s t a n c e b e t w e e n P o i n t sDimensionalityFigure 1.Distances among 100k points gen-erated at random in a unit hypercube [2].From the above perspective,we noticed that we should distinguish indistinctive nearest neighbors from distinctive ones in order to achieve efficient similarity retrieval of multimedia information.If we detect the distinctiveness of the nearest neighbor,we can change the behavior of the NN search to reduce the search cost and to provide more information for users.To this aim,we devised a new NN search algorithm:distinctiveness-sensitive nearest-neighbor search.This algorithm determines the distinctive-ness of the nearest neighbor based on the intrinsic dimen-sionality during search operation.If the nearest neighbor is determined to be indistinctive,the search quits and returns a partial result.This enables us not only to cut down search cost but also to notify users whether the nearest neighbor is distinctive or not.These advantages are especially benefi-cial to the interactive retrieval system which requires quick response as well as informative results.The contribution of this paper is threefold:(1)The notion of distinctiveness is introduced to nearestneighbors.Through theoretical and experimental evaluation,we demonstrate how important it is to distinguish indis-tinctive nearest neighbors from distinctive ones.(2)A way of detecting the distinctiveness of the nearestneighbor is presented.We devised a probabilistic method which determines the distinctiveness of the nearest neighbor.(3)A new type of NN search method,i.e.,distinctiveness-sensitive nearest-neighbor search,is presented.We devised a new NN search method which detects the distinctiveness of nearest neighbors during search op-eration.The benefit of the proposed method is demon-strated through experimental evaluation.This paper is organized as follows.Section 2shows how indistinctive nearest neighbors affect the similarity retrieval of multimedia information.Section 3describes how to es-timate the distinctiveness of nearest neighbors.Section 4Figure 2.Case of 64dimensions of Figure 1.presents the distinctiveness-sensitive NN search algorithm.The results of experiments are shown in Section 5.Section 6contains conclusions.2.Distinctiveness of NNs in feature space2.1.Distinctiveness of NNs in uniform distributionWhen we use NN search,we expect that found neigh-bors are much closer than the others.However,this intu-ition is sometimes incorrect in high dimensional space.For example,when points are uniformly distributed in a unit hypercube,the distance between two points is almost the same for any combination of two points.Figure 1shows the minimum,the average,and the maximum of the dis-tances among 100,000points generated at random in a unit hypercube.As shown in the figure,the minimum of the dis-tances grows drastically as the dimensionality increases and the ratio of the minimum to the maximum increases up to 24%in 16dimensions,40%in 32dimensions,and 53%in 64dimensions.Thus,the distance to the nearest neighbor is 53%or more of the distance to the farthest point in 64dimensional space (Figure 2).In this case,we can consider the nearest neighbor to be indistinctive ,because the differ-ence between the nearest neighbor and the others is negli-gible,i.e.,the other points are as close to the query point as the nearest neighbor is.From the perspective of the sim-ilarity retrieval,when the nearest neighbor is indistinctive,the nearest neighbor has almost the same similarity with the others and does not have distinctive similarity to the query.As Figure 1shows,indistinctive nearest neighbors are more likely to occur as dimensionality increases.This char-acteristics can be verified by estimating the distance to -th nearest neighbor.When points are distributed uniformly within the hypersphere whose center is the query point,theexpected distance to -th nearest neighboris obtained as follows[4]:(1)where is the dimensionality of the space andis the ra-dius of the hypersphere.Then,the ratio of the-NN00.20.40.60.8151015201Dimensionality0.1E {d }2N N E {d }1N N E {d }1N N −Figure 3.Relative difference between the first and the second nearest neighbor.distance to the -NN distance is obtained as follows[4]:(2)Thus,when points are distributed uniformly around the query point,it is expected that the difference between the -th and -th nearest neighbors decreases as the di-mensionality increases.This implies that indistinctive near-est neighbors are more likely to occur in high dimensional space than in low dimensional space.Equation (2)also indicates that the ratio ofto decreases monotonically as increases.Therefore,the maximum of the ratio between two nearest neighbors is obtained for the first and the second nearest neighbor.FromEquation (2),the expected ratio ofto is ap-proximated as follows:(3)By subtracting one from both sides,we can estimate the relative difference between the first and the second nearest neighbor when the dimensionality is :(4)This equation shows that the relative difference between the first and the second nearest neighbor decreases monotoni-cally as the dimensionality increases (Figure 3).Only 20%difference is expected at 5dimensions,and only 10%at 10dimensions.This equation clearly shows that we could not expect strong distinctiveness of nearest neighbors for high dimensional uniform distribution.2.2.Intrinsic dimensionality of feature spaceAs shown above,indistinctive nearest neighbors are more likely to appear in high dimensional space,and the ex-pected relative difference between the first and the second nearest neighbor is inversely proportional to the dimension-ality of the distribution.However,this does not mean thathigh dimensional feature space is useless.We should note that the discussion above is based on the uniform distribu-tion.If the data distribution over the entire feature space is uniform,we can say that the feature space is useless,but in real applications,the data distribution is not uniform at all.Instead of assuming the uniform distribution to the en-tire space,we should employ the intrinsic dimensionality (or effective dimensionality)which is determined by a lo-cal characteristics of the data distribution[4].For example,when the data distribution is governed by a number of dom-inant dimensions,the intrinsic dimensionality is given by the number of such dominant dimensions.In addition,the intrinsic dimensionality may not be consistent over the data set but vary from one local region to another .In real applications,we can expect that the data distribu-tion is so skewed that the intrinsic dimensionality could be much smaller than the dimensionality of the feature space.Therefore,we might have indistinctive nearest neighbors in one region but could have distinctive ones in another.In a region with low intrinsic dimensionality,we can expect distinctive nearest neighbors,while we cannot expect dis-tinctive ones in a region with high intrinsic dimensionality.Thus,the intrinsic dimensionality is the important clue for estimating the distinctiveness of the nearest neighbor.In Section 3,we present a probabilistic method which deter-mines the distinctiveness of the nearest neighbor based on intrinsic dimensionality.2.3.Harmful effect of indistinctive NNsIndistinctive nearest neighbors have harmful effect on the similarity retrieval with the following respects:NN search performance is degraded.When the nearest neighbor is indistinctive,there ex-ist many points that have almost the same similarity with the nearest neighbor.Since these points are very strong candidates for the nearest neighbor,NN search operation is forced to examine many points before de-termining the true nearest neighbor.This degrades the performance of NN search operation.Less informative result is returned.When the nearest neighbor is indistinctive,NN search operation returns the closest point among many strong candidates that have almost the same similarity with the nearest neighbor.This means that all of the candi-dates have slight difference with each other.It is not informative for users to choose the nearest neighbor from plenty of similar candidates.These effects are extremely harmful to the retrieval systems with human-computer interaction.When the nearest neigh-bor is indistinctive,the system forces users to wait until less informative result is answered with slow response.Thus,it is necessary to handle indistinctive nearest neighbors ap-Figure4.Definition of the indistinctive NN.0.20.40.60.8151015200.10.9DimensionalityRejectionProbabilityFigure5.Rejection probability( and).propriately in order to achieve efficient similarity retrievalof multimedia information.3.Estimating the distinctiveness of NNs basedon intrinsic dimensionality3.1.Determination of indistinctive NNsAs mentioned above,the intrinsic dimensionality playsan important role in determining the distinctiveness of thenearest neighbor.Therefore,we devised a probabilisticmethod which determines the distinctiveness of the nearestneighbor based on the intrinsic dimensionality.In thefirst place,we coined a definition of indistinctivenearest neighbors as follows:Definition1Let be the distance from the query point to a nearest neighbor.Then,the nearest neighbor is indis-tinctive if more than or equal to points exist within therange of to from the query point. Here,and are controllable parameters (Figure4).determines the proximity of the query pointand determines the congestion of the proximity.Forexample,we set1.84to and48to at the experimentdescribed in Section5.As dimensionality increases,we are more likely to haveplenty of points in the similar distance with the nearestneighbor.This causes congestion in the proximity of thequery point.The above definition determines indistinctivenearest neighbors by detecting the congestion in the prox-imity.This characteristic can be clearly stated by estimat-ing the probability of congestion.We call this probability the rejection probability.When points are distributed uni-formly in a local region with the intrinsic dimensionality, the probability that points exist in the proximity speci-fied by is obtained as follows:or more points in(5) According to the definition above,the nearest neighbor is indistinctive when points exist in the proximity specified by.Therefore,Equation(5)corresponds to the probabil-ity that the nearest neighbor is regarded as being indistinc-tive when the intrinsic dimensionality is.This probability increases monotonically as the intrinsic dimensionality in-creases.Equation(5)clearly shows that the congestion is more likely to occur when the intrinsic dimensionality is high.Figure5shows the rejection probability when is 1.84471and is48.3.2.Finding appropriate setting of the parametersandThe parameters and play an essential role in the definition of the indistinctive nearest neighbor.We present here a way tofind the appropriate setting of and. As Equation(5)shows,the rejection probability is the func-tion of the intrinsic dimensionality,and the form of the function is controlled by the parameters and.The parameter setting presented below consists of two steps.In thefirst step,we choose what intrinsic dimensionality to be regarded as having indistinctive nearest neighbors.In the second step,we locate two control points in order to control the form of the function,i.e.,the cut-off point and the rejection point.Then,the parameters and are determined from these four parameters.In thefirst step,we choose what intrinsic dimensionality to be regarded as having indistinctive nearest neighbors.We call this dimensionality the rejection dimensionality.We can choose it basing on the expected relative difference de-scribed in Section2.1.As shown in Equation(4)and Figure 3,the expected relative difference between thefirst and the second nearest neighbor is inversely proportional to the in-trinsic dimensionality.Only20%difference is expected at 5dimensions,and only10%at10dimensions.Basing on this expected relative difference,we can choose the rejec-tion dimensionality.For example,if we don’t care20%dif-ference,we choose5for the rejection dimensionality;if we don’t care10%difference,we choose10for the rejection dimensionality.In the second step,we locate two control points in order to control the form of the function.Since we have cho-sen the rejection dimensionality in thefirst step,the ideal form of the function is the step function whose value is zero for the dimensionality less than the rejection dimensional-ity and one for the dimensionality greater than or equal to the rejection dimensionality.However,the composition ofDimensionalityR e j e c t i o n P r o b a b i l i t yTransition Band Pass Band (distinctive)Stop Band (indistinctive)Figure 6.Controlling the re-jection probability.1 1.31861 1.621132 1.41441 3.323263 1.51957 6.863864 1.6533216.02565 1.8447148.02776 2.15959232.4327 2.795513070.998 4.67486525245921.8437DimensionalityR e j e c t i o n P r o b a b i l i t y(a)Parametersand(b)Rejection probabilityFigure 7.The parameters and the rejection probability fordifferent (,,and ).this form is infeasible since it is a discontinuous function.Therefore,we must choose a practical form.We locate two control points by analogy with the low pass filter:the cut-off pointand the rejection point (Figure 6).Here,is the rejection dimensionality chosen in the first step andis the rejection probability at ,while is the cut-off dimensionality and is the rejection prob-ability at (and ).The range of corresponds to the pass band where the intrinsic dimensionality is regarded as having distinc-tive nearest neighbors.The range ofcorresponds to the stop band where the intrinsic dimensionality is re-garded as having indistinctive nearest neighbors.The rangeofcorresponds to the transition band.Once we locate two control points,we can determine the parame-tersand by solving the following simultaneous equa-tions:(6)(7)By the elimination ofthe following equation is obtained:(8)This equation cannot be arithmetically solved.However,since the left side of the equation increases monotonically asincreases,it can be easily solved with numerical meth-ods,e.g.,Newton’s method.Onceis determined,is obtained fromas follows:(9)Now,by Equation (8)and (9),we can determine andfrom the two control points and .As shown above,we can determine the parameters and by locating two control points.The remaining ques-tion is where the control points should be located.The an-swer is that it is a trade-off between the sensitivity and thelocality.By putting the cut-off dimensionality near to the rejection dimensionality ,we can have steep slope in thetransition band (i.e.,).However,this causes the increase inand widens the region to be examined for determining the distinctiveness of the nearest neighbor.Asshown in the following example,increases significantly as approaches .Whenis too large,we cannot ex-ploit a local characteristics of the data distribution.Ideally,we should have both high sensitivity and high locality.But we need to find a compromise between the sensitivity and the locality as shown in the following example.3.3.Sample setting of the parametersandHere,we show an example of the parameter setting ac-cording to the procedure described above.In the first step,we choose the rejection dimensionality based on the ex-pected relative difference between the first and the second nearest neighbor.We choose 10for the rejection dimension-ality since only 10%difference is expected.In the second step,we locate two control points with making a compro-mise between the sensitivity and the locality.Firstly,we need to choose the rejection probability at the cut-off andthe rejection dimensionality,i.e.,and respectively.Here,we use 0.1forand 0.9for .Secondly,we need to choose the cut-off dimensionality .Figure 7shows the parameters and the rejection probability obtained for differ-ent .As shown in Figure 7(b),we have high sensitiv-ity whenis close to .However,as shown in Figure 7(a),the parametersand increase significantly as approaches .Therefore,we need to find a compromisebetween the sensitivity and the locality.Whenis 7,is 2.79551and is 3070.99.This means that the conges-tion is detected only when 3071or more points exist in the proximity of the query point.Apparently,3071is too large;the locality of the proximity specified byis very low.Whenis 5,the selectivity is still high and and are quite small.Therefore,we choose 5for .Finally,the two control points are located at (5,0.1)and (10,0.9)and and are obtained as 1.84471and 48.Figure 5shows the rejection probability with the setting described here.InFigure8.Structure of the R-tree.Section5,this parameter setting is used for the experimen-tal evaluation.4.Distinctiveness-sensitive NN search for mul-tidimensional index structuresIn order to circumvent the harmful effect of indistinc-tive nearest neighbors,we developed a new NN search al-gorithm for multidimensional index structures with apply-ing the distinctiveness estimation method described in the previous section.The algorithm tests the distinctiveness of the nearest neighbor in the course of search operation. Then,when itfinds the nearest neighbor to be indistinc-tive,it quits the search and returns a partial result.We call this NN search as the Distinctiveness-Sensitive Nearest-Neighbor Search since it is sensitive to the distinctiveness of the nearest neighbor.This algorithm not only enables us to determine the distinctiveness of the nearest neighbor but also enables us to cut down the search cost as shown in the following section.4.1.NN search algorithms for multidimensional in-dex structuresThe distinctiveness-sensitive NN search algorithm is de-signed for multidimensional index structures that splits the data space into the nested hierarchy of regions,e.g., the SS-tree[5],the V AMSplit R-tree[6],the X-tree[7],the SR-tree[2],the LSD-tree[8],the Hybrid tree[9],the IQ-tree[10],etc.Figure8illustrates the hierarchical structure of the R-tree.Each node of the tree structure corresponds to a region of the data space.Non-internal nodes store data points,while internal nodes store the information on child nodes,i.e.,region specifications(e.g.,location,size,shape, etc.)and pointers to child nodes.The hierarchical structure permits NN search operation to be completed with examin-ing only some part of the feature space.This reduces CPU time and disk I/O of NN search operation.Therefore,these index structures are widely used for the acceleration of the similarity retrieval of multimedia information.We designed the new algorithm with extending the NN search algorithm presented by Hjaltason et al.[11](This al-gorithm will be called the basic NN search algorithm here-after in order to be distinguished from the distinctiveness-sensitive NN search algorithm).The basic NN search al-Region 1Figure9.NN search with multidimensional in-dex structure.gorithmfinds the nearest neighbor of a given query point as follows.The search starts from the root node of the in-dex structure.At every internal node,the distances from the query point to the child nodes are computed and the child nodes are enqueued into a priority queue in ascending order of the distance.Then,one node is dequeued from the pri-ority queue and the search proceeds to the dequeued node. Thus,nodes are visited in ascending order of the distance from the query point.For example,in Figure9,regions will be searched in the order of Region1,Region2,and Region 3.On the other hand,every time a non-internal node is vis-ited,the distances from the query point to the data points in the node are computed.If it is thefirst visit of a non-internal node,the closest point in the node is chosen for the candi-date of the nearest neighbor;otherwise,the closest point in the node is compared with the latest candidate and the closer one is chosen for the new candidate.The distance to the candidate gives the upper bound of the distance to the nearest neighbor,because the nearest neighbor cannot be farther from the query point than the candidate is.On the other hand,the distance to thefirst node in the priority queue gives the lower bound of the distance to forthcom-ing candidates,because thefirst node of the priority queue is the closest among such nodes that are not visited so far. For example,Figure9illustrates the case where Region1 is visited and the closest point in Region1is chosen for the candidate.Region2and Region3are still in the priority queue.The candidate gives the upper bound of the distance to the nearest neighbor,while the distance to thefirst node of the priority queue,i.e.,the distance to Region2,gives the lower bound of the distance to forthcoming candidates. The search continues until the lower bound of the distance to forthcoming candidates exceeds the upper bound of the distance to the nearest neighbor,i.e.,until no node in the priority queue is closer to the query point than the candidate is.Finally,the nearest neighbor is obtained as thefinal can-didate.Although the description above is the searchfinding thefirst nearest neighbor,it can be easily extended to the( j −1 )( j )( k )( 1 )( 2 )Rank of NNExact NN Exact NNExact NN NN CandidateNN CandidateExact Nearest Neighborsthe TerminationCandidates on indistinctiveFigure 10.Search result of the distinctive-ness-sensitive NN search.-nearest-neighbor search,i.e.,findingnearest neighbors.4.2.Distinctiveness-sensitive NN search algorithmThe distinctiveness-sensitive NN search algorithm is an extension of the basic NN search algorithm described above.The main idea of the extension is counting the num-ber of points that are located within the proximity of the query point in the course of search operation.The prox-imity is speci fied by the parameteras in De finition 1.If the congestion of the proximity is detected,the search is terminated and a partial result is returned.The conges-tion is determined by the parameteras in De finition 1,i.e.,the case where the number of points in the proxim-ity exceedsis regarded as the congestion.In order to count points in the proximity,the algorithm takes advan-tage of the lower bound and the upper bound of the dis-tance to the nearest neighbor.As stated in the de-scription of the basic NN search,the distance to the first node in the priority queue and the distance to the candi-date of the nearest neighbor gives the lower and the up-per bound of(Figure 9).These bounds also give the bounds of the proximity.According to the de finition of the proximity in De finition 1,the range of the proxim-ity is.When the lower and the upper bound of is and respectively,the lower bound and the upper bound of the range of the proximitycan be obtained asand .Sinceis obtained only at the end of the search,the algorithm usesand to count points in the proximity.Becauseand ,the number of points in theproximity is more than or equal to the number of points thatare located in the range ofto .Therefore,the congestion can be detected during the search operation by testing whether the number of points locatedin the range ofto is greater than or equal to .With employing this congestion detection method,thedistinctiveness-sensitive NN search algorithm tests the dis-tinctiveness of the nearest neighbor in the course of search operation.Then,when it finds the nearest neighbor to be in-distinctive,it quits the search and returns the partial result as shown in Figure 10.When -th nearest neighbor is found to be indistinctive during -nearest neighbor search,the search is terminated and candidates on the termination are returned for -th to -th nearest neighbors.Thus,the mixture of the exact nearest neighbors and the nearest neighbor candidates is returned when the indistinctive nearest neighbor is found during the search operation.This algorithm not only en-ables us to determine the distinctiveness of nearest neigh-bors but also enables us to cut down the search cost since this algorithm avoids pursuing exact answers for indistinc-tive nearest neighbors.It should be noted that the distinctiveness-sensitive NN search is different from the approximate NN-search algo-rithms [12,13].The approximate NN search [12]termi-nates when the ratio of the upper bound to the lower boundofis reduced to less than or equal to .Here is the controllable parameter of an error bound.At this point,the candidate is an approximate answer whose error in the distance from the query point is less than or equal to .Thus determines the precision of an approximate answer.The PAC NN search (probably approximately correct NN search)[13]also finds an approximate answer but estimatesthe lower bound offrom the distance distribution of the dataset under the speci fied probability.Thus,the PAC NN search returns an approximate answer under the given precision and the given probability.While these algorithms focus on the precision of an answer,the distinctiveness-sensitive NN search algorithm focuses on the distinctive-ness of the nearest neighbor.As shown in Figure 10,the proposed algorithm returns an inexact answer only when the nearest neighbor is determined to be indistinctive.As long as the nearest neighbor is distinctive,the algorithm returns the exact answer.As its name implies,the salient feature of the proposed algorithm is the sensitivity to the distinctive-ness of the nearest neighbor.5.Experimental evaluation5.1.Evaluation with synthetic datasetsIn order to evaluate the characteristics of the distinc-tiveness-sensitive NN search,we synthesized datasets hav-ing various intrinsic dimensionality.A dataset having the intrinsic dimensionality is synthesized by generating -dimensional points in accordance with the fol-lowing rule:(10)。