IIR02-智能信息检索-杜小勇

vec(qcc) |
Drawbacks of the Boolean Model
• Exact matching • No ranking: • Awkward: Information need has to be translated into a Boolean expression • Too simple: The Boolean queries formulated by the users are most often too simplistic • Unsatisfiable Results: The Boolean model frequently returns either too few or too many documents in response to a user query
IR使用的三类数据
• On-page Data:
– text – multimedia: image/videlo
• Link data/semantic data • Uasge data
– Social network data
What is IR?
• Definition
– IR = <D, Q, R(qi,dj)>, where
• idf factor, the inverse document frequency
– TF*IDF formular: wij = tf(i,j) * idf(i)
The Vector Model
• Let,
– N be the total number of docs in the collection – ni be the number of docs which contain ki – freq(i,j) raw frequency of ki within dj
– common sets of index terms – sharing of weighted terms – likelihood of relevance
• Each set of premises leads to a distinct IR model
The Boolean Model
– – – – – – – Media-related operators Indexing IE Classification and clustering Link analysis Relevance evaluation ……
• Applications
课程安排
• 见word文件
What is Information Retrieval?
– vec(dj) = (w1j, w2j, …, wtj) : a weighted vector associated with the document dj – gi(vec(dj)) = wij : a function which returns the weight of term ki in document dj
智能信息检索
杜小勇教授,中国人民大学 文继荣教授,微软亚洲研究院
About course
• • • • 与MSRA合作,每2年一次. 2008,2010,2012春季学期 以MSRA研究员的系列讲座 属于”开阔视野型”课程
Content
• • • • IR: Concepts and Models System architecture Evaluation and benchmark Key techniques
• A normalized tf factor is given by
– tf(i,j) = freq(i,j) / max(freq(l,j)) – where kl ∈ dj
sim(dj,q)=?
两大类的模型
• 基于文本内容的检索模型
– – – – – – – – 布尔模型 向量空间模型 概率模型 统计语言模型 基于协同的模型 基于链接分析的模型 基于关联的模型 Usage-data
• 与检索对象内容无关的其他检索模型
Classical IR Models ---- Basic Concepts
The Boolean Model
Ka Kb (1,1,0) (1,1,1)
• q = ka (kb kc)
(1,0,0)
Kc
• sim(q,dj) = 1 if
(vec(qcc) /in vec(qdnf)) (ki, gi(vec(dj)) = gi(vec(qcc))) 0 otherwise
– wij >= 0 : a weight associated with (ki,dj) The weight wij quantifies the importance of the index term for describing the document contents
• wij = 0 indicates that term does not belong to doc
j dj

q i
• Sim(q,dj) = cos() = [vec(dj) vec(q)] / (|dj| * |q|) = [ wij * wiq] / (|dj| * |q|) • Since wij > 0 and wiq > 0, 0 <= sim(q,dj) <=1 • A document is retrieved even if it matches the query terms only partially
– q = ka (kb kc) – vec(qdnf) = (1,1,1) (1,1,0) (1,0,0) – vec(qcc) = (1,1,0) is a conjunctive component
Outline
• • • • Boolean Model（BM） Vector Space Model（VSM） Probabilistic Model（PM） Language Model(LM)
Classical IR Models - Basic Concepts
• A ranking is an ordering of the documents retrieved that (hopefully) reflects the relevance of the documents to the user query • A ranking is based on fundamental premises regarding the notion of relevance, such as:
– quantification of intra-document contents (similarity)
• tf factor, the term frequency within a document
– quantification of inter-documents separation (dissimilarity)
The Vector Model
• Sim(q,dj) = [ wij * wiq] / ( |dj| * |q|) • The KEY is to compute the weights wij and wiq ? • A good weight must take into account two effects:
The Vector Model
• Define:
– wij > 0 whenever ki dj – wiq >= 0 associated with the pair (ki,q) – vec(dj) = (w1j, w2j, ..., wtj) vec(q) = (w1q, w2q, ..., wtq) – index terms are assumed to occur independently within the documents ,That means the vector space is orthonormal.
• Bag-of-Word Model • Each document represented by a set of representative keywords or index terms • The importance of the index terms is represented by weights associated to them • Let
Models
IR Model
• Representation
How to represent document/query – Bag-of-word – Sequence-of-word – Link of documents – Semantic Network
• Similarity/relevance Evaluation
• The t terms formபைடு நூலகம்an orthonormal basis for a tdimensional space • In this space, queries and documents are represented as weighted vectors
The Vector Model
• Simple model based on set theory • Queries specified as boolean expressions
– precise semantics – neat formalism – q = ka (kb kc)
• Terms are either present or absent. Thus, wij ∈ {0,1} • Consider