Semantic-based Composite Document Ranking

Semantic-based Composite Document RankingChunchen Liu, Jianqiang LiNEC Laboratories ChinaBeijing, China{liu_chunchen, li_jianqiang}@Abstract—The traditional inform ation retrieval techniques m ainly em ploy statistics of words in docum ent text and/or the link structures of document sets to rank, which have been used successfully in the global web search. However, they produce unsatisfied results for Enterprise search (ES), because ES is very different from Web search. This paper proposes a novel rank approach fitting for the ES environment. With the support of an ontology describing prior knowledge about the target domain, we first mine semantic information (concepts and relations between them) from queries (docum ents) with which to understand the query intentions (docum ent contents) and exploit them for evaluating the query-document relevance; and then the semantic linkages between docum ents are built and consum ed for evaluating the docum ent im portance; finally, the above two evaluations are integrated to produce the final ranking list. Experim ents show that our approach results in significant improvements over existing solutions.Keywords-document ranking; semantic information; enterprise search; enterprise information retrievalI.I NTRODUCTIONEnterprise search (ES) gets a hypergrowth in recent years as the explosive extension of enterprise data. However, when employing the traditional rank models in the ES setting, it soon became clear that they performed poorly for the following reasons: (1) the highly professional and purposive search tasks supported by ES decide that it has a higher accuracy requirement than the Web search does, but the traditional keyword-based relevance computing techniques have intrinsic deficiency in Precision and Recall [1]. (2) Some researchers incorporate semantic information as an effective tool for relevance computing optimization [2~4]. However, they mainly utilize ontology concepts recognized from query (document) for relevance scoring, where the semantic relations defined in ontology are only used for semantic query expansion or semantic similarity calculation, with the role of relations in helping query intention (document content) understanding and correctly matching not fully used. E.g., given a query "Tom hanks movie", several interpretations are reasonable such as "movie d irected by Tom hanks", "movie that Tom hanks acts in", "movie produced by Tom hanks", but which interpretation is the one in the user's mind? With the help of relations, we can match documents with queries more properly in the semantic level. (3) Unlike a set of hyperlinks relate pages together in the web, there are no such linkage relationship between documents in the ES environment, which makes the existing hyperlink-based importance computing solutions [5~6] not applicable.This paper proposes a semantic-based composite rank model (SCRD) meant for the exploitation of full-fledged domain knowledge base to support document ranking. With respect to other strategies, our model not only uses concepts, but also mines the implicit relations between them from queries (documents) and employs them for query (document) understanding and relevance computing. In addition, with the hyperlink missing, SCRD develops a novel method to compute the document importance which will be used in conjunction with the document relevance to further improve the accuracy of rank results. Experiment results show that SCRD performs better on rank accuracy compared with the existing models.II.S EMANTIC-BASED C OMPOSITE D OCUMENT R ANKINGDomainTargetto beUser inputsFigure 1. SCRD rank processOur proposed approach (Fig.1) comprises three stages: (1) Query-document relevance computing; (2) Document importance computing; (3) Composite document ranking. In the first stage, the relevance of each document in the candidate doc set to the query is evaluated by comparing their contained semantic concepts and relations. In succession, the linkages between target documents and reference documents are built and consumed for scoring the document importance. In the last stage, the two factors acquired above are combined for ranking. A.Query-document Relevance ComputingDomainTargetFigure 2. score R computing processEvaluating the relevance between a query and a document by analyzing their contained semantic relations, in other words,2012 IEEE Sixth International Conference on Semantic Computingcomputing the relation-based relevance score, score R , for each candidate document, is the core of our approach. Fig. 2 depicts how to acquire score R , which comprises five steps. 1) Query Concept RecognitionThis step adopts our technique introduced in [7] to map query keywords to concepts in the domain ontology. It first maps each keyword to a set of candidate concepts, and then for each query concept combination, a penalty value is calculated by analyzing the semantic graphs that represent the combination, finally, the top-1 combination with the minimum penalty value is chosen with its contained concepts act as the query concepts.2) Query Intention UnderstandingThis step tries to construct a set of query semantic graphs , S q ={(G q , ηq )| 0≤ηq ≤ }, based on the query concept set C query and the domain ontology to imitate all possible query intentions. A query semantic graph G q is a weighted connected graph, each concept C i ęC query is mapped to a node of G q , the edge between node C i , C j represents a semantic path linking C i with C j in the ontology, with a parameter recording the concepts appearing in the path and a weight ωij denoting the importance of the edge ( its value is the path length). ηq denotes the possibility of the intention described by G q to be just the one in the user ’s mind. The method to acquire S q is as follows. ķMinimum query semantic graph construction. We extract the minimum query semantic graph from the ontology, which is a sub-graph with the minimum number of edges to link all query concept nodes together. (Due to the space limitation, the detailed algorithm is not introduced here.). The number of edges in the sub-graph is denoted as W g-min . ĸ S q acquiring based on W g-min . F irstly, we construct a weighted graph G all (V all ,E all ). Each query concept is a node in V all . Each semantic path path (C i ,C j ) linking C i with C j in the ontology (C i ,C j ę V all ) is mapped to an edge linking C i and C j in E all , with the edge weight parameter (ω) set as lengh (path (C i ,C j )) and the record parameter set as the concept set on the path. When finding semantic paths between C i and C j , make sure that all concepts belonging to V all {C i , C j } shouldn ’t appear in the path. One thing for attention, when two paths are inverse path, only one should be added to E all , in addition, the taxonomy relation in ontology is considered as a unidirectional relation when constructing G all . Secondly, we extract S q from G all (V all ,E all ). A spanning tree of G all (V all ,E all ) with the sum of its edge weights satisfies W g-min ≤W g ≤W g-min +d is a query semantic graph , and its ηq =111n g i i W W (n =|S q |), then all thefound spanning trees comprise S q . d = 2 in our implementation. 3) Document Concept RecognitionThe approach showed in [8], which not only recognizes but also disambiguates concepts, is adopted by this step for annotating document concepts.4) Document Content UnderstandingThis step constructs a set of d ocument semantic graphs , S d ={(G d ,ηd )| 0≤ηd ≤ }, for a document to imitate all possible query intentions covered by it. Let C doc be the set of concepts identified from a document. A d ocument semantic graph G d (V d ,E d ) is a weighted connected graph. Each concept C i ęC doc ģC query is a node in V d . Each edge e (C i C j )ęE d represents a semantic path linking C i with C j in the ontology, with a weight λij denoting the importance of the edge and a parameterrecording the concepts on the path. To insure the run time efficiency, we construct S d offline before the ranking process. The detailed method is as follows.ķConstructing a document semantic graph set S sub for each C sub . (C sub C doc , 2≤|C sub |≤d , d is set to 5 in our implementation).The way to build S sub is nearly same to that to build S q for C query . The only difference is that when finding semantic paths between concepts, make sure that all concepts in the semantic paths must belong to C doc . ĸ Acquiring S d based on {(C sub ,S sub )}. When a query comes, we first compare the query concept set C query with all the document concept subsets C sub . The C sub that satisfies C sub = C doc ģC query is chosen. If C sub == C query , then its corresponding S sub is chosen as S d directly, or else, S sub should be modified by resetting the graph weight for each graph it contains (denoted as G d ), which is realized by first finding all the super graphs of G d from S q , denoted as S'={G q (V q ,E q )| G q (V q ,E q )ęS q && G d is a sub graph of G q }, and then changing the graph weight of G d from ηd to 'min ((||1)(||1))q G S q d q V V K u .5) score R ComputingAfter acquiring the query semantic graph set S q ={(G q , ηq )| 0≤ηq ≤ } and the document semantic graph set S d ={(G d , ηd )| 0≤ηd ≤ }, score R can be computed using Formula (1).score R = d d q qd G S qG S K K ¦¦(1)When score R is obtained, we apply the vector-space model [9] on top of ontology concepts to calculate the concept-based relevance score score C . Then, the combined relevance score of a document with a query is measured as r =G score R +(1-G ) score C (0d G d 1).B. Document Importance ComputingAs the traditional importance computing algorithms are not applicable in the ES environment, we propose a novel approach that evaluates the document importance with the help of external document set acting as implicit knowledge base. Its realization consists of two main steps: (1) The class-instance linkages between target docs and external docs are built by comparing their content, aiming at finding to what extent a target document is supported by external documents (a target document is a class, an external document can be labeled as an instance of a class). F or realization, we adopt our fully automatic text categorization approach FACT [10] for linkage building. (2) Linkage consuming. This step uses the number of instance documents in the external document database to compute the importance of the target document: ķF or each target document d i , we count the number of instantiated external documents n (d i ).ĸAssuming target document d j has the highest number of instance documents max , its importance is H ; and the importance score of each other target document d i is measured as score I (i )=H ×n (d i )/max . H ę[0,1] is the maximum value a target can have.C. Composite Document RankingBoth the query-document relevance and the document importance are used to produce the final ranking list, with an idea that when documents have approximate relevance degreeto the query, the more important a document is, the upper position it should be put on. The ranking process is as follows: ķBy adopting the query-document relevance score as the only feature, the target documents are clustered into k groups. ĸFor each group, computing the average value of the relevance scores of the contained documents, denoted as S1. The sequence of these groups is decided by S1. ĹUsing document importance score to re-rank documents in each group. k is a factor balancing the query-document relevance and the document importance. We adopted the improved k-means algorithm [11] to cluster target documents.III. E XPERIMENT AND E VALUATIONAn English dataset (Internet Movie Database, IMDB) and a Chinese dataset (Vehicle repair dataset, VR) are chosen to test our model. We acquire IMDB via public available plain text exports ( ) and parse it into IMDB ontology serving as the semantic knowledge base, which contains 12 classes, 16 types of relations, 7,568,079 class instances and 18,944,925 relation instances. The homepages of 100,000 movies are downloaded randomly from as the target doc set. About 2,000,000 movie reviews are also downloaded serving as the external doc set. VR is our internal dataset acquired from the Toyota company. The vehicle repair ontology serves as the semantic knowledge base, which contains 2 classes, 3 types of relations, 27,963 class instances and 45,724 relation instances. The 6,109 symptom records in VR comprise the target doc set, and the 424,370 actual repair cases comprise the external doc set. To evaluate the document relevance, we perform a user study where 12 participants are asked to indicate the two-point Likert scale.To evaluate the performance of SCRD, we adopt the Precision and Recall metrics that are widely used. In addition, the rank sequence number comparison is also carried out. We perform the F -feature evaluation on the two test datasets to select the optimal parameter of G = {0, 0.05, 0.1, 0.15, ..., 1}. G =0.6 (G =0.55) produces the highest F value on IMDB (VR). To make sure that SCRD is not biased, we set their averaged value 0.575 as the final parameter value.A. Precision and Recall ComparisonTwo versions of SCRD are to be test. Version1 considers document relevance only for ranking, and Version2 employs our complete composite rank method where document importance is also incorporated to rank. The concept-based ranking approach [2] is utilized as baseline, which has been proved to gain much improvement on rank accuracy compared with the keyword-based search engines.F ig. 3 illustrates the Recall and Precision comparison results. When testing on IMDB, we run rank models on two target document sets separately: the homepage set (F ig.3(a)) and the review set (Fig.3(b)). Each document in the former set is well organized and written, which contains all-around and complete information about a movie. However, documents in the latter set are written by cinephile showing their viewingexperiences, which only involve part of the movie information.(a) Results on homepage set in IMDB(b) Results on review set in IMDB(c) Results on VRFigure 3. Recall and Precision comparisonAs Fig. 3(a) shows that our relevance-only ranking method (Version1) outperforms the baseline on precision about 20% on average and 28% at most when recall is 0.9, which illustrates that the semantic relation between concepts is an important factor that can't be ignored to help to match queries with documents in a more accurate way. However, the role of document importance in improving precision is not so obvious, and Version2 only outperforms the relevance only approach about 1.2% on average. After analyzing Fig. 3(b), we find that when searching on the review set, the baseline works much worse than it does on the homepage set (a fall of 9.9% on average) demonstrating that it is sensitive to the lacking of information. But our approach is more stable with precision reduced only by 3.8% on average, in other words, our relevance computing method is more robust than the baseline. F ig. 3(c) shows the comparison on VR. As we can see, the performance of Version1 on VR is much worse than it does on IMDB, where only 2.6% optimization on precision is achieved. That's because the semantic relations between concepts in the knowledge base of VR are so simple where only one relation links two concepts together in most cases, which leads score R to make weak contribution to distinguish docs. In conclusion, our relevance-only rank method is more robust and can generate more accurate results than the baseline does, especially when the relations between concepts are complex.B.Rank Sequence Number ComparisonTABLE I. R ANK S EQUENCE C ONPARISON ON IMDB Position expert baseline ver1ver2d exp, base d exp,ver1d exp,ver21 2 3 4 5 6 7 8 91011121314151617181920 12486591071415123131116171820193131110129812415141617185197620132134678121159101514161719182012345678910111213141516171819202117264195102134711211141125212253103723121141122244101411Avg.error 6.1 2.6 1.9 TABLE II. R ANK S EQUENCE C OMPARISON ON VR Position expert baseline ver1ver2d exp, base d exp,ver1d exp,ver21 2 3 4 5 6 7 8 91011121314151617181920 123465101498711121315181620171935426101516114131278119201719186451231171213159141081618192017123456789101112131415161718192023125528661554943215223421735822372213211362411121221Avg.error 3.7 3.3 1.4We choose one sample query randomly from each dataset, and compare the rank sequence of test models on them with that of experts in detail. The top-20 documents from the rank of Version2 are chosen as target ones to be re-ranked by experts and other models. Tableĉ and Ċ show the results. As we can see, in our experiments on both datasets, the rank sequence acquired by SCRD (Version2) is more close to that by the experts than the baseline does. Speaking concretely, according to the expert in tableĉ, the average ranking error for the baseline is 6.1, but it is 1.9 for SCRD. According to the expert in tableĊ, the average ranking error for the baseline is 3.7, but it is reduced to 1.4 by SCRD. In addition, based on the comparison on Version1 and Version2, we can see that although document importance takes unobvious effect in improving rank precision and recall, it plays a crucial role in putting documents in a more proper sequence, which reduces average errors effectively (1.9 on VR and 0.7 on IMDB). This validates that our importance calculating and utilizing method is reasonable. C.Time ComplexityThe most time consuming parts of our method are the query concept recognition, query intent understanding and the composite document rank processes whose performance affect the effectiveness of SCRD directly (table ċ). The computer we use is a windows-based PC with a Pentium(R) D CPU at 2.80GHz and 1 Gbytes of RAM. As we can see, in general, the promising results (36.2ms at most) over nearly 0.2 million documents demonstrate the feasibility of SCRD.TABLE III. E XPERIMENT R ESULTS OF T IME C OMPLEXITY ON B OTHIMDB AND VRq uerykeywordsquery concept recognitionquery intent understanding(ms)compositedocumentrank (ms)total delaysof SC RD(ms)1 0.1 8.9 9.72 1.2 9.5 11.53 3.1 11.3 15.14 8.5 11.1 20.35 23.2 12.2 36.2IV.C ONCLUSIONThis paper proposes SCRD for document ranking in ES environment. With the support of domain ontology, the semantic information implied in query (document), which is crucial for query intention understanding (document content understanding), is extracted and consumed for document relevance computing. Document importance is measured with the help of external doc sets through a semantic linkage-based approach, and then it is employed in conjunction with the document relevance for acquiring the final ranking list.R EFERENCES[1]J. Lee, J. Min and C.W. Chung, "An effective semantic search techniqueusing ontology," Proc. of the 18th Int'l Conf. World Wide Web, pp.1057-1058, 2009.[2]P. Cstells, M. F ernandez and D. Vallet, "An adaptation of the vector-space model for ontology-based information retrieval," IEEE Trans. on Knowledge and Data Engineering, vol.19, no.2, pp.261-272, 2007.[3]T. Hao and Z. Lu, "Categorizing and ranking search e ngine’s result bysemantic similarity," Proc.of the 2nd Int’l Conf. on Ubiquitous Information Management and Conmmunication, pp.284-288, 2008.[4]M. Daoud and M. Boughanem, "A personalized graph-based documentranking model using a semantic profile," Proc. of the 18th Int’l Conf. on User Modeling, Adaption and Personalization, pp.171-182, 2010.[5]S. Brin and L. Page, "The anatomy of large-scale hypertextual websearch engine," Proc. of the 7th Int'l Conf. World Wide Web, pp.107-117, 1998.[6]J. Kleinberg, "Authoritative sources in a hyperlinked environment," Proc.of ACM-SIAM Symposium on Discrete Algorithms, pp.668-677, 1998. [7]J.Q. Li, C.C. Liu, Y, Zhao and B. Liu, "A practical system for semanticinformation retrieval," unpublished.[8]F. Brauer, M. Huber and G. Hackenbroich, "Graph-based conceptidentification and disambiguation for enterprise search," Proc. of the 19th Int'l Conf. World Wide Web, pp.171-180, 2010.[9]G. Dalton and M. McGill, Introduction to Modern Information Retrieval,McGraw-Hill,1983.[10]J.Q. Li, Y. Zhao and B. Liu, "F ully automatic text categorization byexploiting WordNet," Proc. of the 5th Asia Information Retrieval Symposium, pp.1-12, 2009.[11]Z. Wang, G.Q. Liu and E.H. Chen, "A k-means algorithm based onoptimized initial center points," Journal of Pattern Recognition and Artificial Intelligence, vol.22, no.2, pp.299-304, 2009.。