PAT04-projection methods - 2015

9
Transform
6
4
2
0 -6 -4 -2 -2 0 2 4 6 8
• This gives us a new set of axes defined by the eigenvectors • The new coordinate system is created from the first by rotating the datapoints
4
Computing the variance along a line • First zero-mean the datapoints • The position of a point along direction u (a unit vector) is given by u
xu x cos θ x u
13
PCA using SVD • Use the idea of Principal Components Analysis directly on the datapoints
– The first two or three components of PCA will provide a good representation of the data xT 0 • Data matrix X from the datapoints X xT – The points must have zero mean p
10
Transform
6
4
2
0 -6 -4 -2 -2 0 2 4 6 8
Σ RΛR
Σ' ?
T
-4
-6
10 8 6 4 2 0 -10 -5 -2 -4 -6 -8 -10 0 5 10
11
Transform • The variance of the new components are T T Var ( xu ) u Σu u u u u • The covariance matrix of the transformed data, ’ can be found by and has the form
– And so on, if we have more than two dimensions
8
Transform
6
4
2
0 -6 -4 -2 -2 0 2 4 6 8
• This gives us a new set of axes defined by the eigenvectors • The new coordinate system is created from the first by rotating the datapoints
1 X 'T X ' 1 RT XT XR RT R n n
It is diagonal
0 0
0
1
0
0 12Exam Nhomakorabeale• Example
6 4 2
0 -6 -4 -2 -2 0 2 4 6 8
8.307818 4.131241 4.131241 7.648128 0 12.12236, e 0 (0.73471,0.67838)T
• Linear algebra gives us a solution to this problem via the eigenvalues and eigenvectors of the covariance matrix • The first eigenvector is in the direction of maximum variance, (u=e0 maximises uTu) • The second eigenvector is orthogonal to the first, and in the direction of maximum variance, subject to that constraint
x
xu
• The variance of the points along this axis can be written
1 1 2 Var( xu ) xu ,i (xi u)2 n i n i
5
Computing the variance along a line • First zero-mean the datapoints • The position of a point along direction u (a unit vector) is given by u
Projection methods
将数据投影到一个特定方向，… 确定这个方向的标准是什么？ 如何用PCA/SVD确定这个方向？ PCA/SVD有何缺陷？ 如何用LDA确定这个方向？
1
Projection Methods Consider this two-class dataset
14
1
12
0.8
10
0.6 0.4
8
0.2 0 0 -0.2 5 10 15 20
6
4
-0.4 -0.6
2
-0.8
0 0 2 4 6 8 10 12 14
-1
By projecting along a particular axis, we can perform an efficient 1D classification. There is an implied linear decision boundary in the original space. A number of ways to do this, but the process involves • Find a transformation that optimises a criterion • Reduce the number of dimensions
xu x cos θ x u
x
xu
• The variance of the points along this axis can be written
• So the problem is to find the maximum value of
1 1 1 T T 2 2 Var( xu ) xu ,k (xk u) u X Xu uT Σu n k n k n
– The rotation matrix is the matrix of eigenvectors
-4
-6
R e0
e1
10 8 6 4 2 0 -10 -5 -2 -4 -6 -8 -10 0 5 10
• • • •
original data point x new data point x’=? original data matrix X new data matrix X’=?
u Σu
6
T
u Σu
• When u is ____, it is maximized; its maximal value is _______.
T
7
Maximisation
6
4
2
0 -6 -4 -2 -2 0 2 4 6 8
-4
-6
10 8 6 4 2 0 -10 -5 -2 -4 -6 -8 -10 0 5 10
2
14
12
10
8
6
4
2
0 0 2 4 6 8 10 12 14
how to find the axis? optimize what?
3
PCA • PCA – Principal components analysis
– Choose directions along the maximum variation in the dataset • The variation is measured by the
– Maximise the ratio of between-class to within-class Large between-class variation and withinclass variation Smaller within-class variation
-4
-6
1 3.83359, e1 (0.67838,0.73470)T
10 8 6 4 2 0
-10
-5 -2 -4 -6 -8 -10
0
5
10
0.73471 - 0.67838 R 0.67838 0.73470 0 12.12236 0 3.83359
T 2 S T
therefore ： R V 2 E S / n
• The rotation matrices are the same • The eigenvalues are the square of the singular values divided by n
17
• which axis is the result of PCA? • which one do we desire?
8
14
12
10
6
4
2
0 0 2 4 6 8 10 12 14
(co)variance of the datapoints • The direction of maximum variance may provide the most information about the spread of points
Eigendecomposition RE RT