CS276InformationRetrievalandWebSearchChristopherM：cs276信息检索和WebsearchchristopherM

▪ Theorem: Exists an eigen decomposition
diagonal
▪ (cf. matrix diagonalization theorem)
▪ Columns of U are the eigenvectors of S
Unique for
distinct eigenvalues
▪ If
is a symmetric matrix:
▪ Theorem: There exists a (unique) eigen
decomposition S = QLQT
▪ where Q is orthogonal:
▪ Q-1= QT
▪ Columns of Q are normalized eigenvectors
All eigenvalues of a real symmetric matrix are real.
for complex l, if S - lI = 0 and S = ST Þ l Î Â
All eigenvalues of a positive semidefinite matrix are non-negative
▪ Diagonal elements of are eigenvalues of
Introduction to Information Retrieval
Sec. 18.1
Diagonal decomposition: why/how
Let U have the eigenvectors as columns:
instead of
æ 60ö èççç860ø÷÷÷
we would get æ60ö èççç800ø÷÷÷
▪ These vectors are similar (in cosine similarity, etc.)
Introduction to Information Retrieval
Introduction to Information Retrieval
Eigenvalues & Eigenvectors
▪ Eigenvectors (for a square mm matrix S)
Sec. 18.1
(right) eigenvector eigenvalue
Example
▪ How many eigenvalues are there at most?
only has a non-zero solution if This is a mth order equation in λ which can have at most m distinct solutions (roots of the characteristic
Eigenvalues & Eigenvectors
Sec. 18.1
For symmetric matrices, eigenvectors for distinct eigenvalues are orthogonal
Sv{1,2} l = v , {1,2} {1,2} and l1 ¹ l2 Þ v1 · v2 = 0
Introduction to Information Retrieval
Sec. 18.1
Matrix-vector multiplication
▪ Thus a matrix-vector multiplication such as Sx (S, x as in the previous slide) can be rewritten in terms of the eigenvalues/vectors:
Introduction to Information Retrieval
Sec. 18.1
Matrix-vector multiplication
▪ Suggestion: the effect of “small” eigenvalues is small. ▪ If we ignored the smallest eigenvalue (1), then
Thus SU=U, or U–1SU=
And S=UU–1.
Introduction to Information Retrieval
Sec. 18.1
Diagonal decomposition - example
Recall
é S =ê
2
ë1
1 2
ù
ú;l1 = 1, l2 = 3.
û
æ The eigenvectors ç
polynomial) – can be complex even though S is real.
Introduction to Information Retrieval
Matrix-vector multiplication
Sec. 18.1
é30 0 0ù
S
=
ê ê
0
20 0úú
ëê 0 0 1ûú
has eigenvalues 30, 20, 1 with corresponding eigenvectors
1 v1 0
0
0 v2 1 0
0 v3 0 1
On each eigenvector, S acts as a multiple of the identity matrix: but as a different multiple on each. Any vector (say x= ) æççca24nö÷÷be viewed as a combination of the eigenvectors: èç 6 ø÷x = 2v1 + 4v2 + 6v3
2 ùúéê 1
0 ùúéê 1/
2
-1 /
2
ù ú
ëê -1/ 2 1 / 2 ûúë 0 3 ûëê 1/ 2 1 / 2 ûú
Q
(Q-1= QT )
Why? Stay tuned …
Introduction to Information Retrieval
Sec. 18.1
Symmetric Eigen Decomposition
▪ Clothes, movies, politics, …
▪ Can we represent the term-document space by a lower dimensional latent space?
Introduction to Information Retrieval
Linear Algebra Background
ù ú
ë -1 1 ûë 0 3 ûë 1/ 2 1 / 2 û
Introduction to Information Retrieval
Swenku.baidu.comc. 18.1
Example continued
Let’s divide U (and multiply U–1) by
2
é Then, S= ê
1/
2
1/
▪ But if you want to dredge, Strang’s Applied Mathematics is a good place to start.
▪ What do these matrices have to do with text?
▪ Recall M N term-document matrices …
è
1 -1
ö ÷
and
æ ç
øè
1 1
ö ÷
form
ø
é U =ê
ë
1 -1
1
ù ú
1û
Inverting, we have
é U -1 = ê
1/2
-1/ 2
ù ú
ë 1/2 1/2 û
Recall UU–1 =1.
é Then, S=UU–1 = ê
1
1
ùé úê
1
0
ùé úê
1/ 2
-1/ 2
"w Î Ân, wTSw ³ 0, then if Sv = lv Þ l ³ 0
Introduction to Information Retrieval
Sec. 18.1
Example
▪ Let
é S =ê
2
1
ù ú
ë1 2û
Real, symmetric.
▪ Then
S
I
2
1
2
1
| S I | (2 )2 1 0.
▪ The eigenvalues are 1 and 3 (nonnegative, real).
▪ The eigenvectors are orthogonal (and real):
æ ç
1
öæ ÷ç
1
ö ÷
è -1 ø è 1 ø
Plug in these values and solve for eigenvectors.
▪ But everything so far needs square matrices – so …
Introduction to Information Retrieval
Similarity Clustering
▪ We can compute the similarity between two document vector representations xi and xj by xixjT
Introduction to Information Retrieval
Introduction to
Information Retrieval
CS276: Information Retrieval and Web Search Christopher Manning and Pandu Nayak Lecture 13: Latent Semantic Indexing
Sx = S(2v1 + 4v2 + 6v3)
Sx = 2Sv1 + 4Sv2 + 6Sv 3= 2l1v1 + 4l2v2 + 6l3v3
Sx = 60v1 + 80v2 + 6v3
▪ Even though x is an arbitrary vector, the action of S on x is determined by the eigenvalues/vectors.
0 1 0 1 1 2 2 2 1 0 1 0 2 3 2 4
Introduction to Information Retrieval
Time out!
▪ I came to this class to learn about text retrieval and mining, not to have my linear algebra past dredged up again …
Then, SU can be written
é
ùé
ê
úê
SU = Sê v1 ... vn ú = ê l1v1 ...
ëê
ûú ëê
ùé úê
lnvn ú = ê v1
ûú ëê
é
ù
ê
ú
U = ê v1 ... vn ú
ëê
ûú
ùé úê
l1
ù ú
... vn úê
...
ú
ûúëêê
ln
ú ûú
▪ Columns are orthogonal.
▪ (everything is real)
Introduction to Information Retrieval
Sec. 18.1
Exercise
▪ Examine the symmetric eigen decomposition, if any, for each of the following matrices:
▪ Let X = [x1 … xN] ▪ Then XXT is a matrix of similarities ▪ XXT is symmetric ▪ So XXT = QΛQT ▪ So we can decompose this similarity space into a set
Introduction to Information Retrieval
Sec. 18.1
Eigen/diagonal Decomposition
▪ Let
be a square matrix with m linearly
independent eigenvectors (a “non-defective” matrix)
Introduction to Information Retrieval
Ch. 18
Today’s topic
Latent Semantic Indexing
▪ Term-document matrices are very large
▪ But the number of topics that people talk about is small (in some sense)