lecture8 Matrix Completion

Σ = Σ1I2
0
0 diag(σ3 · · · σn)
Q0 0I
Q0
QT 0
=
Σ
,
0I
0I
QT 0 0I
where QQT = I. Then we have
UΣVT = U Q 0 Σ QT 0 VT .
0I
0I
In general, for any Ui and Vi that satisfy A = UiΣViT , we can rewrite in the following form as
tr(X−1) − ttr(X−1YX−1) + t2tr((X−1Y)2X−1) − · · · − tr(X−1)
= lim
t↓0
t
= −tr(X−1YX−1)
= −tr(X−2Y)
8.3 Subgradient
Definition 8.2. If function f is convex and proper (which means domf = {x ∈ E|f (x) < ∞} is non-empty), φ is said to be subgradient of f (x) at xˆ, if it satisfies φ, x − xˆ ≤ f (x) − f (xˆ) for all x ∈ E.
Let Z = (zij) denote user i rates movie j with ranking zij ∈ {1, 2, 3, 4, 5}. However, some movies are not rated by any people, i.e. zij is missing. Our purpose is to complete the ranking matrix. Let X be the m × n complete matrix and Ω = {(i, j)|zij is observed}. We can reasonably suppose that X is a low rank matrix. Our purpose is to minimize the error between Z and X with the constraint that X is low rank, i.e.
simply say f is differentiable (on E).
Example 8.1. f (X) = log |X|, X ∈ Sn++, find f (X; Y), where Y ∈ Sn++.
Solution:
log |(X + tY)| − log |X|
f (X, Y) = lim
8-4
There are three examples of Schatten p-norm:
• p = 1, A ∗ =
n i=1
σi.
• p = 2, A F =
n i=1
σi2.
• p = ∞, A ∞ = σ1. We call σ1 the spectrum radius, and A ∞ is also called spectral norm.
f (xˆ; d) = lim
(2)
t↓0
t
when the limit exists. When the directional derivative f (xˆ; d) is actually linear in d, that
is f (xˆ; d) = a, d for some element a of E. Then, we say f is (Gaˆteanx) differentiable at xˆ with (Gˆateanx) derivative ∇f (xˆ) = a. If f is differentiable at every point in E, then we
1 min X2
(zij − xij )2
(i,j)∈Ω
s.t. X a low rank matrix
Since X is a low rank matrix, we cannot merely use Z ≈ UT X as an approximation of Z. We need to add a sparse matrix S, i.e. Z = X + S, for a matrix can always be represented as the sum of a low rank matrix and a sparse matrix. Then our objective function is:
=
X−
1 2
X−
1 2
YX−
1 2
X
1 2
,
we
have
X−1Y
∼
X−
1 2
YX−
1 2
,
i.e.
X−1Y
and
X−
1 2
YX−
1 2
have the same eigenvalues.
Consider that X and Y are positive definite
symmetric
matrices,
then
X−
1 2
=
(X−
1 2
)T
= 0 and for any z = 0, zT Yz > 0.
Hence
zT
X−
1 2
YX−
1 2
z
>
0
and
X−
1 2
YX−
1 2
is
positive
definite.
Example 8.2. f (X) = tr(AT X), A ∈ Rp×m, X ∈ Rp×n, find f (X; Y)
Lemma 8.1. Let A and R be given m × n matrices, · is Schatten p-norm, φ(·) is the corresponding norm on singular vector, then there is a SVD of A such that
Machine Learning
Matrix Completion
Lecture Notes 8: Matrix Completion
Professor: Zhihua Zhang
Scribe:Yuxi Zhang, Ruotian Luo
8 Matrix Completion
8.1 Problem Background
min
X,S
1 2
Z−X−S
2 F
+
λ1
X
∗ + λ2
S
1
(1)
However, this objective function is not differentiable. Therefore, we need to introduce the definition of directional derivative and subgradient.
t↓0
t
log |X(I + tX−1Y)| − log |X|
=| + log |I + tX−1Y| − log |X|
= lim
t↓0
t
log |I + tX−1Y|
= lim
t↓0
t
= lim
n i=1
log
(1
+
tλi
(X−1
Y))
t↓0
t
=
lim
t↓0
n i=1
Proposition 8.1. For any convex proper function f : E → (−∞, ∞), the point xˆ is a global minimizer of f iff the condition 0 ∈ ∂f (xˆ) holds.
Recall the definition of general matrix norms: Definition 8.3. For A ∈ Rm×n, A is a function of A which satisfies the following conditions.
1. A ≥ 0
2. A = 0 iff A = 0
3. A + B ≤ A + B
8-3
4. αA = |α| A According to the definition, it’s easy to derive that matrix norm is convex:
αA + (1 − α)B ≤ α A + (1 − α) B
8-2
Solution:
tr(AT (X + tY)) − tr(AT X)
f (X, Y) = lim
t↓0
t
tr(AT X) + tr(tAT Y)) − tr(AT X)
= lim
t↓0
t
= tr(AT Y)
= A, Y
Example 8.3. f (X) = tr(X−1), find f (X, Y)
Addition condition: Definition 8.4. A matrix norm · is called consistent if:
AB ≤ A B .
We here consider a kind of norm function which satisfy UT AV = A , U,V are orthogonal matrices. UT U = UUT = I and VT V = VVT = I. A = UΣVT , A is m × n matrix, U is m × m matrix; Σ is m × n; V is n × n.