基于距离的索引

17
Indexing: Scope of research
• • • • Tree structure indexes Static operations Off-line construction Range query
18
Category: data type & similarity
1. Data type: One-dimensional, R Similarity measure: Euclidean norm (absolute value of difference) Index: One-dimensional indexing Example: B-tree Data type: Multi-dimensional, Rn Similarity measure: Euclidean norm Index: Multi-dimensional indexing Example: kd-tree Data type: Other type Similarity measure: Other measurement Index: ? Example: ?
Range query R(q,r)
k-Nearest Neighbor query:kNN(q)
Google Map top 10 results
r
q
3
Example 1
• Find all students with score in [75, 85]:
SELECT name FROM student WHERE ABS(score-80)<=5;
14
Index: performance measurements
• Goal: fast data lookup
– Time complexity
• Number of distance calculations • Number of I/O
– Space complexity
• Ideal case: Log or even constant time • Worst case: Sequential scan of database

d(x,y) = d (y,x) (symmetry) d(x, y) >= 0 and d(x, y) = 0 iff x = y (non negativity) d(x,z) <= d(x,y) + d(y,z) (triangle inequality)
x d(x,z)
z d(y,z)
Goal:
– A coarse filter, retrieval a small subset of database as candidate for fine filtering.
Similarity
– Semi-cosine distance
10
Protein sequence homology [BIBE06]
Goal:
– Local alignment
Similarity:
– Global alignment of 6-mers with mPAM matrix (weighted edit distance)
11
More applications
• • • • Spelling checker Sound recognition Face recognition P2P network
4
Example 2: 深圳大学附近的加油站
5
Example 3: 音乐搜索
6
Molecular Biological Information System (MoBIoS)
/~mobios
7
Image retrieval [CIT05]
8
Conserved primer pair [ISMB04]
[Uhlmann 1991 & Yianilos 1993]
•Choose vantage points •Partition the data
VP21 VP1 R1 d(VP1,q) VP22 Case 2. If d(VP1,q) < R1 - r then search inside the sphere

Bounding sphere methods
– – –
22
Hyper-plane methods [Uhlmann 1991]
•Choose centers •Partition the data
C1,C2
Left of L Right of L
C1 C2
L
23
Vantage Point Tree (VPT)
R21 d(VP1, x)≤R1
VP1,R1
d(VP1, x)>R1
VP21,R21 …
VP22,R22 …
d(VP22, x)≤R22 d(VP , x)>R22 Case 1. If d(VP1,q) > R1 +22 r then search outside the sphere
R22
r q Case 3. Bad case: query object close to partition boundary, descend both children
Similarity:
– Hamming distance of 18-mers
9
Mass-spectra coarse filter [Bioinformatics06]
Given:
– A mass-spectra database – A query mass-spectra (high-dim vector)
Given
– A database of sequences – A query sequence
Methodology
1. Break database and query into k-mers 2. Similarity query of k-mers 3. Chain the results.
d(x,y) y
20
How it works?
Range query R(snoppy,2)
Advantages
– Generality
• One-dimensional data • Multi-dimensional data with Euclidean norm • Any metric space ? 1 100
15
Two-step indexing
1. Off-line construction
– Time not limited (too much) ≤O(n2) – Space limitation, < O(n2) – Determine query performance
2. On-line search
Distance-Based Indexing 基于距离的索引
毛睿 深圳大学计算机与软件学院 国家高性能计算中心深圳分中心
Outline
1. 2. 3. 4. Similarity query and applications Indexing for similarity query Distance-based (metric space) indexing The pivot space model
– Time < O(n)
16
Indexing: Methodology
• Recursive partition and pruning
Example: Twenty guesses • Secretly determine a number, x, in a range, say [1, 1,000,000] • Each time, the audience can ask a question with possible answers “yes” or “no” only. And the audience always get the true answer based on x. • To find out x, how many times the audience need to ask questions?
12
Importance of similarity query
• 相似性搜索已经成为多媒体信息系统基于内 容搜索的基本需求，其性能已经成为衡量海 量多媒体系统查询功能的重要指标
– 冯玉才.曹奎.曹忠升. 一种支持快速相似检索的多维索引 结构, 软件学报2002(8): 1678-1685.
• 相似性搜索在整个计算生物学研究任务中所 占比例高达35%
– A uniform programming model
• the distance oracle is given • One index mechanism for most
Disadvantages
– Not fast enough
d(Michael,Linc)=1 d(Linc, Snoopy) = 100
19
2.
3.
3. ห้องสมุดไป่ตู้istance-based indexing
Metric Space 度量空间
a pair, M=(D,d), where D is a set of points d is a [metric] distance function with the following:
99<=d(Michael, Snoopy)<= 101
21
Data partition: three families