线性系统理论-最小二乘法

9 · 38
BIT
×
H.B.Ma
3. Estimation and Least-Squares
Least-Squares
3
Estimation and Least-Squares
assume A is skinny and full rank, so – m > n so we have more measurements than unknowns; equations are overdetermined – null(A) = {0}, so there is at most one solution x to Ax = ymeas usually ymeas = Ax + w with w some error or noise; there are usually no solutions to Ax = ymeas instead find the least-squares solution, the x that minimizes ymeas − Ax
Linear System Theory:
Least Squares Hongbin Ma mathmhb@gmail.com School of Automation Beijing Instititue of Technology October 18, 2012 Beijing, China
Mainly based on S. Lall’s slides Made with mslide package in MTEX.
12 · 38
BIT
×
H.B.Ma
3. Estimation and Least-Squares
Least-Squares
so xopt is optimal if and only if AT Axopt = AT ymeas called the normal equations A is skinny and full rank, so AT A is invertible, so as before xopt = (AT A)−1 AT ymeas
13 · 38
BIT
×
H.B.Ma
3. Estimation and Least-Squares
Least-Squares
Pseudo-Inverse Approach
if A is skinny and full rank then AT A is invertible and A† = (AT A)−1 AT ˆΣ ˆV T , to see this, notice that the thin SVD of A is A = U where V is square and orthogonal, so ˆΣ ˆ 2V T AT A = V U and ˆΣ ˆ −2 V T V Σ ˆU ˆT = V Σ ˆ −1 U ˆ T = A† (AT A)−1 AT = V U
7 · 38
BIT
×
H.B.Ma
2. The Pseudo-Inverse
Least-Squares
Example
the thin svd: −0.49 0.30 35.69 0 = 0.12 −0.91 0 7.02 0.86 0.30 pseudo-inverse: 0.33 0.38 0.31 0.20 † A = 0.79 −0.11 0.39 −0.53 −0.15 −0.73 0.33 0.31 0.79 0.39 −0.15 0.38 0.20 −0.11 −0.53 −0.73
1. The Key Points of This Section
Least-Squares
1
The Key Points of This Section
estimation problems: given ymeas , find the least-squares solution x, that minimizes ymeas − Ax control problems: given ydes , find the minimum-norm x that satisfies ydes = Ax the SVD gives a computational approach it also gives useful information even when important assumptions don.t hold – estimation: usually need A skinny and full rank – control: usually need A fat and full rank it gives us quantitative information about the usefulness of the solutions
5 · 38
BIT
×
H.B.Ma
2. The Pseudo-Inverse
Least-Squares
2
The Pseudo-Inverse
ˆΣ ˆV T the thin SVD is A = U
here ˆ is square, diagonal, positive definite Σ ˆ and V ˆ are skinny, orthonormal columns U the pseudo-inverse of A is ˆΣ ˆ −1 U ˆT A† = V it is computed using the SVD
Properties of The Pseudo-Inverse
if A is invertible, then A† = A−1 A is m × n ⇐ A† is n × m (A† )† = A (AT )† = (A† )T (λA)† = λ−1 A† for λ = 0 caution: in general, (AB )† = B † A†
BIT
×
H.B.Ma
3. Estimation and Least-Squares
Least-Squares
Geometric Approach
pick as estimate xopt ; by orthogonality Axopt − ymeas ⊥ range(A) which holds if and only if Axopt − ymeas → null(AT ) which holds if and only if AT (Axopt − ymeas ) = 0
6 · 38 BIT × H.B.Ma
2. The Pseudo-Inverse
Least-Squares
Example
rank 2 matrix: −5 −5 −14 −8 1 A = −1 0 4 5 4 11 10 24 11 −6 the full svd:
−0.49 0.30 0.82 35.69 0 0 0 0 = 0.12 −0.91 0.41 0 7.02 0 0 0 × 0.86 0.30 0.41 0 0 000 0.33 0.31 0.79 0.39 −0.15 0.38 0.20 −0.11 −0.53 −0.73 0.25 −0.86 0.36 −0.25 0.01 0.45 −0.26 −0.47 0.67 −0.25 0.69 0.22 −0.14 −0.25 0.62
10 · 38
BIT
×
H.B.Ma
3. Estimation and Least-Squares
Least-Squares
Using Differentiation
The residual is r = Ax − ymeas which we would like to minimize. So r
4 · 38
BIT
×
H.B.Ma
1. The Key Points of This Section
Least-Squares
One More
null(AT A) = null(A) also easy via the SVD: A T A = V ΣT U T U ΣV T = V Σ T ΣV T which gives an SVD of AT A. ΣT Σ has the same number of non-zero elements as Σ, so both A and AT A have null space span{vr+1 , . . . , vn }
14 · 38
BIT
×
H.B.Ma
3. Estimation and Least-Squares
Least-Squares
Left-Inverse Property
when A is skinny and full rank, A† is a left-inverse for A A† A = I
OUTLINE
1 The Key Points of This Section 2 The Pseudo-Inverse 3 Estimation and Least-Squares 4 Control and Minimum-Norm Solutions 5 Matrices Without Full Rank 6 Matlab and the Pseudo-Inverse 7 History of Least Squares 3 6 10 23 31 32 33
1 35.69
0
1 7.02
0
−0.49 0.12 0.86 0.30 −0.91 0.30
8 · 38
BIT
×
H.B.Ma
2. The Pseudo-Inverse
Least-Squares
Key Point: Pseudo-Inverse Solves
least-squares estimation problems minimum-norm control problems
2 T = xT AT Ax − 2ymeas Ax + ymeas 2
differentiate with respect to x and set to zero
T 2xT AT A − 2ymeas A=0
so the optimum x is xopt = (AT A)−1 AT ymeas
11 · 38
because A† = (AT A)−1 AT this is exactly what we need for estimation, because if ymeas = Ax, choosing estimate xest = A† y give xest = A† Ax = x
15 · 38
3 · 38
BIT
×
H.B.Ma
1. The Key Points of This Section
Least-Squares
Important Facts
null(AT ) = range(A)⊥ easy via the SVD: because if the SVD of A is A = U ΣV T then range(A) = span{u1 , . . . , ur } also the SVD of AT is A T = V ΣT U T so null(AT ) = span{ur+1 , . . . , un }
BIT
×
H.B.Ma
3. Estimation and Least-Squares
Least-Squares
来自百度文库
Effects of Noise on Estimation
Suppose we measure ymeas = Ax + w and we use estimator B with BA = I . xest = Bymeas If w ≤ 1, then the estimate lies in the ellipsoid xest → {x + Bw| w ≤ 1} because xest = B (Ax + w) = x + Bw. Picking B = A† gives semiaxis directions vi and −1 semiaxis lengths σi , worst error e = 1/σmin (A)