Total least squares

Errors-in-variables (EIV) is a robust modeling technique in statistics, which assumes that every variable can have error or noise. Errors-in-variables is also referred to as total least squares (TLS), in a broad sense, in the literature of computational mathematics and engineering. However, TLS in a strict sense implies the application of EIV or orthogonal regression to a linear model $\mathbf {Ax} =\mathbf {b}$.

Robust linear regression

In linear regression, the least squares (LS) attributes all error to the dependent variables. It has variant versions according to other error configurations, including total least squares (i.e. orthogonal error), data least squares (DLS), and constrained or structured TLS.

Given an observation vector $\mathbf {b} \in \mathbb {R} ^{n}$ and a data matrix $\mathbf {A} \in \mathbb {R} ^{n\times m}$, consider the solution of the overdetermined system of equations

$\mathbf {Ax\approx b}$.

The ordinary least squares method (OLS) yields the solution $\mathbf {x}$ that minimizes the Euclidean norm of the residuals $\|{\mathbf {Ax-b} }\|_{2}$, where $\|\cdot \|_{2}$ is also known as the two-norm. The residual is an estimate of the error. Equivalently, the OLS problem can be paraphrased by

$\min _{\mathbf {x} }\|\Delta \mathbf {b} \|_{2}\quad {\mbox{ subject to }}\quad \mathbf {Ax} =\mathbf {b} +\Delta \mathbf {b} .$

If the data matrix $\mathbf {A}$ is also noisy (i.e. error in both the dependent and the explanatory variables), the OLS solution is no longer optimal. In cases where orthogonal optimization is acceptable, TLS offers a proper formulation:

$\min _{\mathbf {x} }\|{[{\Delta \mathbf {A} \,\Delta \mathbf {b} }]}\|_{F}\quad {\mbox{ subject to }}\quad (\mathbf {A} +\Delta \mathbf {A} )\mathbf {x} =\mathbf {b} +\Delta \mathbf {b} ,$

where $\|{\cdot }\|_{F}$ is the Frobenius norm (or colloquially the "length" of the vector); and the perturbations $\Delta \mathbf {A}$ and $\Delta \mathbf {b}$ are used to compensate for the noisy signals $\mathbf {A}$ and $\mathbf {b}$, respectively. This formulation of TLS also implies that the noises are assumed to be independent and identically distributed (i.i.d.) both in $\mathbf {A}$ and $\mathbf {b}$. Note that the objective can have a weighting matrix according to the distribution of errors if the distribution is known or well-estimated, which is called the constrained or structured TLS.

In the other case, where the noise is only in $\mathbf {A}$, DLS can be used alternatively as

$\min _{\mathbf {x} }\|{[{\Delta \mathbf {A} }]}\|_{F}\quad {\mbox{ subject to }}\quad (\mathbf {A} +\Delta \mathbf {A} )\mathbf {x} =\mathbf {b} .$

The solution of the OLS problem can be obtained by using the (pseudo-)inverse of the data matrix. Solutions to the TLS and DLS problems have been shown to be closely connected to a set of singular vectors of the (augmented) system-related matrix corresponding to the minimum singular value.

References

• S. V. Huffel and P. Lemmerling, Total Least Squares and Errors-in-Variables Modeling: Analysis, Algorithms and Applications. Dordrecht, The Netherlands: Kluwer Academic Publishers, 2002.
• S. Jo and S. W. Kim, "Consistent normalized least mean square filtering with noisy data matrix," IEEE Trans. Signal Processing, vol. 53, no. 6, pp. 2112-2123, Jun. 2005.
• R. D. DeGroat and E. M. Dowling, "The data least squares problem and channel equalization," IEEE Trans. Signal Processing, vol. 41, no. 1, pp. 407–411, Jan. 1993.
• T. Abatzoglou and J. Mendel, "Constrained total least squares," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP’87), Apr. 1987, vol. 12, pp. 1485–1488.