optimizing visual search reranking via pairwise learning

280IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 13, NO. 2, APRIL 2011Optimizing Visual Search Reranking via Pairwise LearningYuan Liu and Tao Mei, Member, IEEEbe merely judged by the text-based approaches, as textual information may fail to precisely describe the visual content. For example, when users search for images with a warm color, the images cannot be easily measured by any textual description. To address this issue, visual search reranking has received increasing attention in recent years [7], [12], [18], [19], [21], [23], [27], [33], [37], [39]. It can be defined as reordering visual documents based on the initial search results or some auxiliary knowledge, aiming to improve search precision [21]. The research on visual search reranking has proceeded along three dimensions from the perspective of how external knowledge is exploited: 1) self-reranking [7], [12], [18], [23], [27], [33], [39], which mainly focuses on detecting relevant patterns (recurrent or dominant patterns) from the initial search results without any external knowledge; 2) example-reranking [21], [37], in which the query examples are provided by users so that the relevant patterns can be discovered from these examples; and 3) crowdreranking [19], which mines relevant patterns from the crowdsourcing knowledge available on the web. The first dimension, i.e., self-reranking, although relies little on the external knowledge, cannot deal with the “ambiguity problem” which is derived from the text queries. Taking the query “jaguar” as an example, the search system cannot determine what the user is really searching for, whether it is “an animal” or “a car.” As illustrated in Fig. 1(a), results with different meanings but all related to “jaguar” can be found in the top-ranked results of “jaguar.” A similar observation can be found in Fig. 1(b), in which diverse images with the surrounding text of “apple” are mixed in the search results of “apple.” To address this problem, the second and the third dimensions leverage some auxiliary knowledge to better understand the query. Specifically, the second dimension, i.e., example-reranking, leverages a few query examples to train the reranking models. However, the typical model-based approaches usually assume the availability of a large collection of training data, which cannot be satisfied as users are reluctant to provide enough query examples while searching. To address the limitation of lack of query examples, the third dimension, i.e., crowd-reranking, leverages crowdsourcing knowledge collected from multiple search engines. It is reported that much higher improvements can be obtained since different engines can inform and complement the relevant visual information for the given query. However, it still cannot avoid the ambiguity problem as current visual search engines mainly support the text query. To summarize, on one hand, in the example-reranking, insufficient user-provided examples will limit the applications as it only works well with a large collection of training data. On the other, self-reranking and crowd-reranking still cannot deal withAbstract—Visual search reranking is defined as reordering visual documents (images or video clips) based on the initial search results or some auxiliary knowledge to improve the search precision. Conventional approaches to visual search reranking empirically take the “classification performance” as the optimization objective, in which each visual document is determined relevant or not, followed by a process of increasing the order of relevant documents. In this paper, we first show that the classification performance fails to produce a globally optimal ranked list, and then we formulate reranking as an optimization problem, in which a ranked list is globally optimal only if any arbitrary two documents in the list are correctly ranked in terms of relevance. This is different from existing approaches which simply classify a document as “relevant” or not. To find the optimal ranked list, we convert the individual documents to “document pairs,” each represented as a “ordinal relation.” Then, we find the optimal document pairs which can maximally preserve the initial rank order while simultaneously keeping the consistency with the auxiliary knowledge mined from query examples and web resources as much as possible. We develop two pairwise reranking methods, difference pairwise reranking (DP-reranking) and exclusion pairwise reranking (EP-reranking), to obtain the relevant relation of each document pair. Finally, a round robin criterion is explored to recover the final ranked list. We conducted comprehensive experiments on an automatic video search task over TRECVID 2005–2007 benchmarks, and showed consistent improvements over text search baseline and other reranking approaches. Index Terms—Optimization, reranking, visual search. pairwise learning, searchI. INTRODUCTION HE proliferation of digital capture devices and the explosive growth of community-contributed media contents have led to a surge of research activity in visual search [16]. Due to the great success of text document retrieval, most existing visual search systems rely entirely on the text associated with the visual documents (images or video clips), such as document title, description, automatic speech recognition (ASR) results from videos, and so on. However, visual relevance cannotManuscript received June 09, 2010; revised September 22, 2010; accepted December 17, 2010. Date of publication January 06, 2011; date of current version March 18, 2011. This work was performed when Y. Liu was visiting Microsoft Research Asia as a research intern. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Andrea Cavallaro. Y. Liu is with the Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei 230027, China (e-mail: tc@). T. Mei is with Microsoft Research Asia, Beijing 100190, China (e-mail: tmei@). Color versions of one or more of the figures in this paper are available online at . Digital Object Identifier 10.1109/TMM.2010.2103931T1520-9210/$26.00 © 2011 IEEELIU AND MEI: OPTIMIZING VISUAL SEARCH RERANKING VIA PAIRWISE LEARNING281TABLE I CLASSIFICATION AND RERANKINGTABLE II PERFORMANCE COMPARISON BETWEEN CLASSIFICATION AND RANKINGFig. 1. Examples of the top-ranked results from a commercial web image search engine: (I) top 30 original web search results; (II) query examples provided by a user; and (III) filtered web examples based on the original web search results and query examples.the ambiguity problem as current visual search engines only support the text query. To address these issues, in this paper, we will leverage both query examples and crowdsourcing knowledge simultaneously. Specifically, we first feed the text query to a visual web search engine and collect the visual documents along with the associated text. To avoid the ambiguity problem, we then use the query examples to filter the web results and get more clean “web examples.” Also taking Fig. 1 as the example, by using the two user-provided query examples, the web examples are filtered based on visual similarity [24] to better catch the user’s search intent. By using the filtered web examples, we will give motivation to directly optimize the ranking list in Section I-A. A. Motivation: Classification versus Ranking Most current approaches to visual search reranking take the “classification performance” as the optimization objective, in which the reranking is formulated as a binary classification problem to determine whether or not a visual document is relevant, and then increase the ranking order of the relevant documents. However, although some systems have obtaineda high search performance, it is known that an optimal classification performance cannot guarantee an optimal search performance [40]. Suppose we have a hypothesis space with the two hypothesis functions, and . The two hypotheses predict a ranking for a query over a document corpus. is the indicator of document : relevant, 0: irrelevant) to the query . By relevance ( using the example shown in Tables I and II, we can demonstrate that models which optimize for classification performance are not directly concerned with the ranking which is often measured by average precision (AP) [34]. When we learn the models which optimize for the classification performance (such as “accuracy” and “recall”), the objective is to learn a “threshold” such that documents scoring higher than the threshold can be classified as relevant and the docu,a ments scoring lower as irrelevant. Specifically, with threshold between the documents 8 and 9 gives two errors (documents 1–2 incorrectly classified as relevant), yielding both an , a threshold accuracy and recall of 0.80. Similarly, with between documents 3 and 4 gives three errors (i.e., documents 7–9 are incorrectly classified as relevant), yielding an accuracy and recall of 0.70 and 0.75, respectively. Therefore, a learning method which optimizes for classification performance would since it results in a higher accuracy and recall. Howchoose ever, this will lead to a suboptimal ranking measured by the AP scores and the number of pairs correctly ranked. In summary, conventional reranking approaches which regard reranking as classification problem fail to give a global optimal ranking list. To address this problem, we can redefine the reranking as guaranteeing the highest probability that each arbitrary document pair is correctly ranked in terms of relevance. Therefore, we consider the visual documents in a pairwise manner, with each pair being represented as ordinal score, and we formulate the reranking as an optimization problem to find an optimal document pair set. Specifically, the objective is to minimize three types of distance, i.e., ranking distance, knowledge distance, and smooth distance. First, the optimal pair set will maximally preserve the initial ranking order, in other words, the distance between initial pair set and reranked pair set (ranking distance) will be minimized. For this, we will give two pairwise distance measurements between two pair sets. Second, the reranked pairs should be consistent with the mined cues as282IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 13, NO. 2, APRIL 2011• We analyze the “ambiguity problem” in the visual search reranking by text query, and introduce a novel approach by leveraging the crowdsourcing knowledge and query examples to avoid these problems. • We theoretically formulate visual search reranking as a global optimization problem by considering the visual documents in a pairwise manner. Furthermore, under the framework, we develop two pairwise reranking methods and give detailed solutions. The remainder of this paper is organized as follows. Section II introduces the related work on visual search reranking. Section III presents the proposed approach. Section IV shows the experiments, followed by the conclusions in Section V. II. RELATED WORKFig. 2. Flowchart of pairwise reranking.much as possible, that is, the knowledge distance should be minimized simultaneously. The mined cues are represented by the concept relatedness, which is learned from the query examples and public crowdsourcing knowledge. Third, smooth distance is explored based on the assumption that if two pairs have similar characteristics (concept relatedness in this paper), then the corresponding ordinal score should be close as well and vice versa. To minimize the three distances simultaneously, we develop two reranking methods, i.e., exclusion pairwise reranking (EPreranking) and difference pairwise reranking (DP-reranking), and provide the solutions. B. Overview The objective of pairwise learning for visual search reranking is to obtain an optimal set of document pairs by mining the query examples and crowdsourcing knowledge. The flowchart is illustrated in Fig. 2, including two main components, i.e., learning and reranking. Given a textual query, an initial ranked list of the visual documents (taking “images” as the example) is obtained by text-based search techniques. Meanwhile, this query is fed to a web search engine to collect a set of image search results along with their surrounding texts. In the learning process, we first use the query examples to filter the web search results and get the relatively clean “web examples” based on visual similarity. We then extract the textual keywords and detect the concept confidences for web examples by means of the pre-trained concept detectors [24], as concepts can be often viewed as an effective representation, like keywords in the text domain [7], [21], [22]. Through analyzing the keywords and the concept confidence scores, we can mine the concept relatedness to the given query. In the reranking process, the initial ranked list is converted to image pairs and represented by the mined concept relatedness. Then, the reranking is formulized as an optimization problem to find an optimal pair set. Finally, the reranked list is recovered from such pair sets based on a “round robin criterion.” This paper makes the following three main contributions. • We argue that reranking should be targeting at a higher probability that any two documents are ranked correctly in terms of relevance, rather than simply classify each individual documents as relevant or not.There exists rich research on visual search reranking in recent years. We first give a brief survey on the related works about image and video search reranking. Additionally, the topics about query-concept mapping for visual search and learning to rank have also been attracting an increasing amount of attention. We also discuss the representative works for theses two topics. A. Visual Search Reranking As aforementioned, the research on image and video search reranking has proceeded along three dimensions from the perspective of the external knowledge used [19]: self-reranking, which requires no external knowledge; example-reranking, which is based on user-provided query examples; and crowd-reranking, which exploits online crowdsourcing knowledge. The first dimension, i.e., self-reranking, also called unsupervised-reranking [21] aims to detect recurrent patterns (viewed as the relevant cues), such as video stories, image patches, and high-level concepts/categories, in the initial search results without any external knowledge [7], [8], [12], [18], [23], [27], [33], [39]. For example, Hsu et al. formulate the reranking process as a random walk over a context graph, where video stories are nodes and the edges between them are weighted by multimodal similarities. The video stories with similar visual appearance and text descriptions are linked compactly, and thus, the documents ranked lower can be picked up if they are strongly linked with considerable top-ranked ones [12]. Fergus et al. first perform visual clustering on initial returned images by probabilistic latent semantic analysis (pLSA), and then learn the visual object category based on the visual clusters; finally, the images are reranked according to the distance to the learned categories [7]. To tackle the “ambiguity problem,” the second and third dimensions explore some auxiliary knowledge to better understand the query. Specifically, the second dimension, i.e., example-reranking, also called supervised-reranking [21], leverages a few query examples to train the reranking models [21], [37] or gives some suggestions to improve the search precision [41]. For example, Yan et al. and Schroff et al. view the query examples as pseudo-positives and the bottom-ranked initial results as pseudo-negatives. A reranking model is then built based on these samples by support vector machine (SVM) [37]. Liu et al. use the query examples to learn the relevant and irrelevantLIU AND MEI: OPTIMIZING VISUAL SEARCH RERANKING VIA PAIRWISE LEARNING283concepts for a given query, and then identify an optimal set of document pairs via an information theory. The final reranking list is directly recovered from this optimal pair set [21]. To leverage more examples, the third dimension, i.e., crowdreranking, uses online crowdsourcing knowledge obtained from public social networks [19]. For example, our recent work first constructs a set of visual words based on local image patches collected from multiple image search engines, explicitly detects the so-called salient and concurrent patterns among the visual words, and then theoretically formalizes the reranking as an optimization problem on the basis of the mined visual patterns [19]. To address the above issues in example-reranking and crowdreranking, in this paper, we will leverage query examples and crowdsourcing knowledge simultaneously in an efficient way.examples to filter the web results and get cleaner “web examples.” By combining the filtered visual web examples and associated text, we aim to obtain more robust related concepts. C. Learning to Rank Learning to rank is now intensively studied in both information retrieval and machine learning communities. It is to automatically create a ranking model by using labeled training data and machine learning techniques. Generally, existing methods for learning to rank fall into three paradigms. For a given query and a set of retrieved documents, pointwise approaches try to directly estimate the relevance label for each query-document pair. While one may show that these approaches are consistent for a variety of performance measures [4], they ignore relative information within collections of documents. Pairwise approaches, as proposed by [1], [11], and [13], take the relative nature of the scores into account by comparing pairs of documents. They ensure that we obtain the correct order of documents even in the case when we may not be able to obtain a good estimate of the ratings directly. Finally, listwise approaches, as proposed by [2], [9], [32], and [36], treat the ranking in its totality. In learning, it takes ranked lists as instances and trains a ranking function through the minimization of a listwise loss function defined on the predicted list and the ground truth list [36]. Each of those paradigms focuses on a different aspect of the dataset while largely ignoring others. Recently, Moon et al. proposed a new learning to rank algorithm, IntervalRank, by using isotonic regression to balance the tradeoff between the three paradigms [25]. In this paper, we are motivated by the framework of pairwise learning to rank, and present a pairwise reranking method by mining the relationship between pairs of documents in initial ranked list. Considering there is limited number of documents in initial ranked list, the listwise paradigm is not considered in this paper. III. VISUAL SEARCH RERANKING VIA PAIRWISE LEARNING A. Problem Formulation Suppose we have a document set with documents to be reranked, where , and is the th document in the initial ranked list. We convert the initial ranked , list to a pair set denotes that is ranked before in the initial where ranked list. is the size of the pair set , and . Let and denote the initial and the reranked pairwise ordinal ( for short), respectively. and score for the pair reflect the ordinal relation or relevance difference of and , i.e., the higher the score, the higher degree is ranked before . In the reranking problem, there are three factors that should be considered: 1) the initial ranking order should be preserved as this indicates the relevance information from a text perspective; 2) the ordinal relation of document pairs should be consistent with the learnt knowledge (i.e., concept relatedness); and 3) the ordinal relation among the document pairs should be smooth; in other words, a similar existing concept will lead to a similar ordinal relation (i.e., smooth assumption). Therefore, we canB. Query-Concept Mapping for Visual Search Visual search with a set of high-level concept detectors has attracted increasing attention in recent years [14], [17], [21], [22], [28], [31], [35]. Intuitively, if queries can be automatically mapped to related concepts, search performance will benefit significantly. For example, the “face” concept can benefit people-related queries and the “sky” concept can also be high weighted for outdoor-related queries. Motivated by these observations, the problem of discovering related concepts, also called “query-concept mapping,” has been focused on by many researchers recently. For example, Kennedy et al. mine the top-ranked and bottom-ranked search results to discover related concepts by measuring the mutual information [14]. The basic idea is that if a concept has high mutual information with the top-ranked results and low mutual information with the bottom-ranked results, it will be considered as a related concept. Avoiding the ambiguity problem, Li et al. and Liu et al. leverage a few query examples to find related concepts; specifically, Li et al. use the tf-idf-like scheme [17] and Liu et al. explore the mutual information measurement [21]. Both methods are motivated by the information-theoretic point of view, that is, the more query examples bear more information of a concept, the more the concept will be related to the corresponding query. These methods mentioned above, however, only leverage the visual information extracted from either the top-ranked results or the query examples. The other types of information, such as the text, are entirely neglected. To solve this problem, Wang et al. linearly combine the text and visual information extracted from the text query and visual examples, respectively [35]. However, most of the practical text queries are very short, often represented by one or two words or phrases, so it is difficult to obtain robust concept relatedness information from such a small amount of text. Besides, the ambiguous problem also cannot be avoided, for instance, the query “jaguar” may be related to both an “animal” and a “car,” but the two concepts have little relation to each other. In this paper, we first feed the text query to a public visual web search engine and collect the visual documents along with the associated text; to avoid the ambiguity problem, we use query284IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 13, NO. 2, APRIL 2011TABLE III DIFFERENT RERANKING PROCESSES WHEN PARAMETERS GO TO 0 AND 1formulate the reranking problem by minimizing the following energy function: (1) where and are -dimensional vectors, and is an -dimensional vector, with each element indicating the importance or relatedness of a concept to the given query, where is the size of the concept lexicon. corresponds to the ranking distance between the initial ranked list and reranked list, corresponds to the knowledge distance between the reranked list corresponds to the smooth and the learnt concepts, and distance among the document pairs in reranked list. The parameters , , and tune the contribution of the three distances, , , , . In Section IV we will and demonstrate that their combination is better than using each one individually. From (1), we can find that when the three parameters are set to 0 and 1, the reranking process will rely on different knowledge. We list the details in Table III. B. Discovery of Concepts Relatedness To discover concept relatedness, we leverage the query examples and online crowdsourcing knowledge. We first feed the text query in a public web search engine to collect web search results; we then use the query examples to filter the web search results via visual similarity and get cleaner “web examples.” Let denote the query example set provided by the user, and is the number of query examples. The final web example set is derived by (2) where is the L1 distance and is the fixed threshold estimated by the average of distances between each query example pair and it is defined as follows:denotes the number of query example pairs. where There are two methods for detecting concept relatedness to the given query, based on a pre-defined concept lexicon. • Use a set of pre-trained concept detectors over web examples. The detectors were trained SVMs over three visual features: color moments on a 5-by-5 grid, an edge distribution histogram, and wavelet textures. The confidence scores of the three SVM models over each visual document are then averaged to give the final concept detection confidence. The details of the features and concept detection can be found in [24], in which a set of concept detectors are built mainly based on the low-level visual features and SVM for “high-level feature detection task.” • Mine the surrounding text of web examples. The standard stemming and stop word removal [24] are first performed as a pre-process; then terms with the highest frequency are selected to form a keyword set and match the concepts in the lexicon; here Google Distance (GD) [3] is adopted to measure two textual words:(4) and are the numbers of images conwhere and , respectively, and is taining words and . These the number of images containing both numbers can be obtained by performing a search by textual words on the Google image search engine [10]. is the total number of images indexed in the Google search is used to avoid the extremely engine. The operation large value. We can see that GD is a measure of semantic interrelatedness derived from the number of hits returned by the Google search engine for a given set of keywords. Keywords with the same or similar meanings in a natural language sense tend to be “close” in the units of GD, while the words with dissimilar meanings tend to be departed far away from each other. If the two search terms never occur together on the same web page, but do occur separately, then the GD between them is infinite. If both terms always occur together, then their GD is zero. By combining the above two methods, the relatedness of the th concept to a given query, i.e., , is given by(5)(3)where is the confidence score of the concept of obtained from the pre-trained concept dethe web example tectors. is a parameter to tune the contribution of concept detectors and surrounding text. Empirically, a relatively lower would be more suitable for the concept detector with limited performance. More details about how to set will be discussed in Section IV. Note that all elements in should be normalized to .LIU AND MEI: OPTIMIZING VISUAL SEARCH RERANKING VIA PAIRWISE LEARNING285C. Distance Definitions This section discusses the three distances in (1), i.e., ranking , knowledge distance , and smooth distance distance . Unlike traditional ranking distances, which are calculated between two ranking lists by using the relevance scores of the “individual documents” [19], [33], our ranking distance is calculated by using the ordinal scores of the “document pairs” after converting the ranking list to a pair set. The following two strategies of ranking distances are introduced: • Ranking distance I: difference square (6) • Ranking distance II: accumulated exclusion (7) where is an operator often called “XOR,” determining whether the right and the left values are consistent or not . Let be the ordinal score for the document pair ; the function is defined as (8) The knowledge distance and smooth distance are defined by means of a set of high-level concepts. First, the document pairs with in the pair set are represented by a matrix each row indicating the concept relatedness of a document pair, denotes the relatedness of the th pair to the i.e., th concept , and is defined by the following logistic function [20]: (9) A logistic function is a common sigmoid curve. It can model the “S-shaped” curve (abbreviated S-curve) of growth of some population. The initial stage of growth is approximately exponential; then, as saturation begins, the growth slows, and at maturity, approaches 1 when growth stops. We can see is much higher than , while it approaches 0 when ) is much lower than . When is equal to , . Then following [19], the knowledge distance is defined as (10) denote the vector with entries . can be viewed as the approximate cosine similarity between the concept-based representation of document and the learned concept relatedness of the given query pair , since and belong to the range of . Cosine similarity is a measure of similarity between two vectors of n dimensions by finding the cosine of the angle between them. Based on the assumption that if the pair and have similar concept relatedness, then the corresponding ordinal score and Letshould be close as well and vice versa, the smooth distance is defined as (11) which is widely used in semi-supervised learning methods [42], [43], where is the similarity between and represented as the vector difference of concept relatedness; it is computed by using the Gaussian kernel (12) where is the scaling parameter, and it is often set by the median value of all distances between each two pairs and . D. Pair Optimization Based on the distance defined in Section III-C, we integrate them into (1), and have the following two kinds of objective reranking functions in the matrix way: • Reranking function using ranking distance I(13) We call this optimization problem as difference pairwise reranking (DP-reranking). We can obtain the solution of (13) as follows (proven in the Appendix): (14) where is an identity matrix whose diagonal elements are 1 , where and the others are 0. and is a diagonal matrix with its - -element . and are obtained by replacing the last eleis obtained by ment of and with zero, respectively. . replacing the last row of with Letting , we can see that (14) consists and . They correspond to the initial of two parts, i.e., search results and learnt concept relatedness, respectively, and both are smoothed by each other. Therefore, the reranking list can be viewed as the combination of the initial search results and the learnt external knowledge. , Let denote the initial ordinal score for the pair and . It can be given by which is made up of document the normalized order difference between the two documents (15) We can see that , as is ranked before in as defined in Section III-A • Reranking function using ranking distance II: see (16) at the bottom of the next page. We call this optimization problem exclusion pairwise reranking (EP-reranking). The。