Kosambi–Karhunen–Loève theorem

In the theory of stochastic processes, the Karhunen–Loève theorem (named after Kari Karhunen and Michel Loève) is a representation of a stochastic process as an infinite linear combination of orthogonal functions, analogous to a Fourier series representation of a function on a bounded interval. Stochastic processes given by infinite series of this form were considered earlier by Damodar Dharmananda Kosambi.

In contrast to a Fourier series where the coefficients are real numbers and the expansion basis consists of sinusoidal functions (that is, sine and cosine functions), the coefficients in the Karhunen–Loève theorem are random variables and the expansion basis depends on the process. In fact, the orthogonal basis functions used in this representation are determined by the covariance function of the process. If we regard a stochastic process as a random function F, that is, one in which the random value is a function on an interval [a, b], then this theorem can be considered as a random orthonormal expansion of F.

In the case of a centered stochastic process {X_t}_{t ∈ [a, b]} (where centered means that the expectations E(X_t) are defined and equal to 0 for all values of the parameter t in [a, b]) satisfying a technical continuity condition, X_t admits a decomposition

$\mathbf {X} _{t}=\sum _{k=1}^{\infty }\mathbf {Z} _{k}e_{k}(t)$

where Z_{k are pairwise uncorrelated random variables and the functions ek are continuous real-valued functions on [a, b] which are pairwise orthogonal in L²[a, b]. The general case of a process which is not centered can be represented by expanding the expectation function (which is a non-random function) in the basis ek .}

Moreover, if the process is Gaussian, then the random variables Z_k are Gaussian and stochastically independent. This result generalizes the Karhunen–Loève transform. An important example of a centered real stochastic process on [0,1] is the Wiener process; the Karhunen–Loève theorem can be used to provide a canonical orthogonal representation for it. In this case the expansion consists of sinusoidal functions.

The above expansion into uncorrelated random variables is also known as the Karhunen–Loève expansion or Karhunen–Loève decomposition. The empirical version (i.e., with the coefficients computed from a sample) is known as the Karhunen–Loève transform (KLT), principal component analysis, proper orthogonal decomposition (POD), or the Hotelling transform.

Formulation

We will formulate the result in terms of complex-valued stochastic processes. The results apply to real-valued processes without modification by recognizing that the complex conjugate of a real number is the number itself.

If X and Y are random variables, the inner product is defined by

$\langle \mathbf {X} |\mathbf {Y} \rangle =\operatorname {E} (\mathbf {X^{*}} \mathbf {Y} )$

where * represents complex conjugation.

Second order statistics

The inner product is defined if both X and Y have finite second moments, or equivalently, if they are both square integrable. Note that the inner product is related to covariance and correlation. In particular, for random variables of mean zero, covariance and inner product coincide. The autocovariance function K_XX is

${\begin{aligned}K_{\mathrm {XX} }(t,s)&=\operatorname {Cov} [X(t),X(s)]=\langle \mathbf {X} _{t}|\mathbf {X} _{s}\rangle &=\mathrm {E} \{[X(t)-\mu _{X}(t)]^{*}[X(s)-\mu _{X}(s)]\}\\&=\mathrm {E} \{X^{*}(t)X(s)\}-\mu _{X}^{*}(t)\mu _{X}(s)\\&=R_{\mathrm {XX} }(t,s)-\mu _{X}^{*}(t)\mu _{X}(s).\end{aligned}}$

If {X_t}_t is a centered process, then

$\mu _{X}(t)=0\,$

for all t. Thus, the autocovariance K_XX is identical to the autocorrelation R_XX:

$K_{\mathrm {XX} }(t,s)=R_{\mathrm {XX} }(t,s).\,$

Note that if {X_t}_t is centered and t₁, ≤ t₂, ≤... ≤ t_N are points in [a, b], then

$\sum _{k,\ell }\operatorname {Cov} _{\mathbf {X} }(t_{k},t_{\ell })=\operatorname {Var} \left(\sum _{k=1}^{N}\mathbf {X} _{k}\right)\geq 0.$

Statement of the theorem

Theorem. Consider a centered stochastic process {X_t}_t indexed by t in the interval [a, b] with covariance function Cov_X. Suppose the covariance function Cov_X(t,s) is jointly continuous in t, s. Then Cov_X can be regarded as a positive definite kernel and so by Mercer's theorem, the corresponding integral operator T on L²[a,b] (relative to Lebesgue measure on [a,b]) has an orthonormal basis of eigenvectors. Let {e_i}_i be the eigenvectors of T corresponding to non-zero eigenvalues and

$\mathbf {Z} _{i}=\int _{a}^{b}\mathbf {X} _{t}e_{i}(t)\,dt.$

Then Z_i are centered orthogonal random variables and

$\mathbf {X} _{t}=\sum _{i=1}^{\infty }e_{i}(t)\mathbf {Z} _{i}$

where the convergence is in the mean and is uniform in t. Moreover

$\operatorname {Var} (\mathbf {Z} _{i})=\operatorname {E} (\mathbf {Z} _{i}^{2})=\lambda _{i}.$

where λ_i is the eigenvalue corresponding to the eigenvector e_i.

Cauchy sums

In the statement of the theorem, the integral defining Z_i, can be defined as the limit in the mean of Cauchy sums of random variables:

$\sum _{k=0}^{\ell -1}\mathbf {X} _{\xi _{k}}e_{i}(\xi _{k})(t_{k+1}-t_{k}),$

where

$a=t_{0}\leq \xi _{0}\leq t_{1}\leq \cdots \leq \xi _{\ell -1}\leq t_{n}=b.$

Special case: Gaussian distribution

Since the limit in the mean of jointly Gaussian random variables is jointly Gaussian, and jointly Gaussian random (centered) variables are independent if and only if they are orthogonal, we can also conclude:

Theorem. The variables Z_i have a joint Gaussian distribution and are stochastically independent if the original process {X_t}_t is Gaussian.

In the gaussian case, since the variables Z_i are independent, we can say more:

$\lim _{N\rightarrow \infty }\sum _{i=1}^{N}e_{i}(t)\mathbf {Z} _{i}(\omega )=\mathbf {X} _{t}(\omega )$

almost surely.

Note that by generalizations of Mercer's theorem we can replace the interval [a, b] with other compact spaces C and Lebesgue measure on [a, b] with a Borel measure whose support is C.

The Wiener process

There are numerous equivalent characterizations of the Wiener process which is a mathematical formalization of Brownian motion. Here we regard it as the centered standard Gaussian process W(t) with covariance function

$\mathrm {K} _{\mathrm {WW} }(t,s)=\operatorname {Cov} (W(t),W(s))=\min(s,t).$

The eigenvectors of the covariance kernel are easily determined. These are

$e_{k}(t)={\sqrt {2}}\sin \left(k-{\frac {1}{2}}\right)\pi t$

and the corresponding eigenvalues are

$\lambda _{k}={\frac {4}{(2k-1)^{2}\pi ^{2}}}.$

This gives the following representation of the Wiener process:

Theorem. There is a sequence {Z_i}_i of independent Gaussian random variables with mean zero and variance 1 such that

$\mathbf {W} _{t}={\sqrt {2}}\sum _{k=1}^{\infty }\mathbf {Z} _{k}{\frac {\sin \left(k-{\frac {1}{2}}\right)\pi t}{\left(k-{\frac {1}{2}}\right)\pi }}.$

Note that this representation is only valid for $t\in [0,1].$ On larger intervals, the increments are not independent. Convergence is uniform in t and in the L² norm, that is

$\operatorname {E} \left(\mathbf {W} _{t}-{\sqrt {2}}\sum _{k=1}^{n}\mathbf {Z} _{k}{\frac {\sin \left(k-{\frac {1}{2}}\right)\pi t}{\left(k-{\frac {1}{2}}\right)\pi }}\right)^{2}\rightarrow 0$

uniformly in t.

The Brownian bridge

Similarly the brownian bridge $B_{t}=W_{t}-tW_{1}$ which is a stochastic process with covariance

${\textrm {Cov}}(B_{t},B_{s})=\min(t,s)-ts$

can be represented as the series

$B_{t}=\sum _{k=1}^{\infty }Z_{k}{\frac {{\sqrt {2}}\sin(k\pi t)}{k\pi }}$

Applications

Adaptive optics systems sometimes use K–L functions to reconstruct wave-front phase information (Dai 1996, JOSA A).

See also

• Principal component analysis
• Proper orthogonal decomposition
• Polynomial chaos

References

• Guikhman, I.; Skorokhod, A. (1977). Introduction a la Théorie des Processus Aléatoires. Éditions MIR.
• Simon, B. (1979). Functional Integration and Quantum Physics. Academic Press.
• Karhunen, Kari (1947). "Über lineare Methoden in der Wahrscheinlichkeitsrechnung". Ann. Acad. Sci. Fennicae. Ser. A. I. Math.-Phys. 37: 1–79.
• Loève, M. (1978). Probability theory. Vol. II, 4th ed. Graduate Texts in Mathematics. Vol. 46. Springer-Verlag. ISBN 0-387-90262-7.
• Dai, G. (1996). "Modal wave-front reconstruction with Zernike polynomials and Karhunen–Loeve functions". JOSA A. 13 (6): 1218. Bibcode:1996JOSAA..13.1218D. doi:10.1364/JOSAA.13.001218.
• Jorgensen, Palle E. T.; Song, Myung-Sin (2007). "Entropy Encoding, Hilbert Space and Karhunen-Loeve Transforms". arXiv:math-ph/0701056. {{cite arXiv}}: Unknown parameter |bibcode= ignored (help)
• Mathar, Richard J. (2008). "Karhunen-Loeve basis functions of Kolmogorov turbulence in the sphere". Baltic Astronomy. 17 (3/4): 383–398. Bibcode:2008BaltA..17..383M.
• Mathar, Richard J. (2009). "Modal decomposition of the von-Karman covariance of atmospheric turbulence in the circular entrance pupil". arXiv:0911.4710 [astro-ph.IM]. {{cite arXiv}}: Unknown parameter |bibcode= ignored (help)
• Mathar, Richard J. (2010). "Karhunen-Loeve basis of Kolmogorov phase screens covering a rectangular stripe". Waves in Random and Complex Media. 20 (1): 23–35. Bibcode:2020WRCM...20...23M. doi:10.1080/17455030903369677.

External links

• Mathematica KarhunenLoeveDecomposition function