Tikhonov吉洪诺夫正则化

Tikhonov regularization
From Wikipedia, the free encyclopedia
Tikhonov regularization is the most commonly used method of of named for . In , the method is also known as ridge regression . It is related to the for problems.
The standard approach to solve an of given as
,b Ax =
is known as and seeks to minimize the
2
b
Ax -
where •is the . However, the matrix A may be or yielding a non-unique solution. In order to give preference to a particular solution with desirable properties, the regularization term is included in this minimization:
2
2x
b Ax Γ+-
for some suitably chosen Tikhonov matrix , Γ. In many cases, this matrix is chosen as the Γ= I , giving preference to solutions with smaller norms. In other cases, operators ., a or a weighted ) may be used to enforce smoothness if the underlying vector is believed to be mostly continuous. This regularization
improves the conditioning of the problem, thus enabling a numerical solution. An explicit solution, denoted by , is given by:
(
)
b A A A x
T
T
T 1
ˆ-ΓΓ+=
The effect of regularization may be varied via the scale of matrix Γ. For Γ
=
αI , when α = 0 this reduces to the unregularized least squares solution provided
that (A T A)−1 exists.
Contents
•
•
•
•
•
•
•
•
Bayesian interpretation
Although at first the choice of the solution to this regularized problem may look artificial, and indeed the matrix Γseems rather arbitrary, the process can be justified from a . Note that for an ill-posed problem one must necessarily introduce some additional assumptions in order to get a stable solution.
Statistically we might assume that we know that x is a random variable with a . For simplicity we take the mean to be zero and assume that each component is
independent with σx. Our data is also subject to errors, and we take the errors
in b to be also with zero mean and standard deviation σb. Under these assumptions the Tikhonov-regularized solution is the solution given the data
and the a priori distribution of x, according to . The Tikhonov matrix is then Γ=
αI for Tikhonov factor α = σb/ σx.
If the assumption of is replaced by assumptions of and uncorrelatedness of , and still assume zero mean, then the entails that the solution is minimal . Generalized Tikhonov regularization
For general multivariate normal distributions for x and the data error, one can apply a transformation of the variables to reduce to the case above. Equivalently,
one can seek an x to minimize
2
2Q P x x b Ax -+-
where we have used 2
P x to stand for the weighted norm x T Px (cf. the ). In the Bayesian interpretation P is the inverse of b , x 0 is the of x , and Q is the inverse covariance matrix of x . The Tikhonov matrix is then given as a factorization of the matrix Q = ΓT Γ. the ), and is considered a . This generalized problem can be solved explicitly using the formula
(
)
()01
0Ax b P A Q
PA A x T T
-++-
[] Regularization in Hilbert space
Typically discrete linear ill-conditioned problems result as discretization of , and one can formulate Tikhonov regularization in the original infinite dimensional context. In the above we can interpret A as a on , and x and b as elements in the domain and range of A . The operator ΓΓ+T A A *is then a bounded invertible operator.
Relation to singular value decomposition and Wiener filter
With Γ
= αI , this least squares solution can be analyzed in a special way via
the . Given the singular value decomposition of A
T V U A ∑=
with singular values σi , the Tikhonov regularized solution can be expressed as
b VDU x
T =ˆ where D has diagonal values
2
2ασσ+=
i i ii D
and is zero elsewhere. This demonstrates the effect of the Tikhonov parameter
on the of the regularized problem. For the generalized case a similar representation can be derived using a . Finally, it is related to the :
∑==q
i i
i
T i i v b
u f x
1
ˆσ
where the Wiener weights are 2
22
ασσ+=i i i f and q is the of A .
Determination of the Tikhonov factor
The optimal regularization parameter α is usually unknown and often in practical problems is determined by an ad hoc method. A possible approach relies on the Bayesian interpretation described above. Other approaches include the , , , and . proved that the optimal parameter, in the sense of minimizes:
()
()[]
21
2
2
2
ˆT
T
X
I
X X
X I Tr y X RSS
G -+--=
=
αβ
τ
where
is the and τ is the effective number .
Using the previous SVD decomposition, we can simplify the above expression:
()()21
'2
22
21
'∑∑==++
-=
q
i i
i
i
q
i i
i
u
b u u
b u y RSS ασ
α
()21
'2
22
0∑=++
=q
i i
i
i
u
b u RSS RSS ασ
α
and
∑∑==++-=+-=q
i i
q
i i i q m m 12
2
2
1222ασαασστ
Relation to probabilistic formulation
The probabilistic formulation of an introduces (when all uncertainties are Gaussian) a covariance matrix C M representing the a priori uncertainties on the model parameters, and a covariance matrix C D representing the uncertainties on the observed parameters (see, for instance, Tarantola, 2004 ). In the special case when these two matrices are diagonal and isotropic,
and
, and, in this case, the equations of inverse theory reduce to the
equations above, with α = σD / σM .
History
Tikhonov regularization has been invented independently in many different
contexts. It became widely known from its application to integral equations from the work of and D. L. Phillips. Some authors use the term Tikhonov-Phillips regularization . The finite dimensional case was expounded by A. E. Hoerl, who took a statistical approach, and by M. Foster, who interpreted this method as a - filter. Following Hoerl, it is known in the statistical literature as ridge regression .
[] References
•(1943). "Об устойчивости обратных задач [On the stability of inverse problems]". 39 (5): 195–198.
•Tychonoff, A. N. (1963). "О решении некорректно поставленных задач и методе регуляризации [Solution of incorrectly formulated problems and the regularization method]". Doklady Akademii Nauk SSSR151:
501–504.. Translated in Soviet Mathematics4: 1035–1038. •Tychonoff, A. N.; V. Y. Arsenin (1977). Solution of Ill-posed Problems.
Washington: Winston & Sons. .
•Hansen, ., 1998, Rank-deficient and Discrete ill-posed problems, SIAM •Hoerl AE, 1962, Application of ridge analysis to regression problems, Chemical Engineering Progress, 58, 54-59.
•Foster M, 1961, An application of the Wiener-Kolmogorov smoothing theory to matrix inversion, J. SIAM, 9, 387-392
•Phillips DL, 1962, A technique for the numerical solution of certain integral equations of the first kind, J Assoc Comput Mach, 9, 84-97
•Tarantola A, 2004, Inverse Problem Theory (), Society for Industrial and Applied Mathematics,
•Wahba, G, 1990, Spline Models for Observational Data, Society for Industrial and Applied Mathematics。