The efficiency comparisons between OLSE and BLUE in a singular linear model

Abstract

This paper is mainly concerned with the efficiency comparison between OLSE and BLUE in a singular linear model. We define the efficiencies between OLSE and BLUE by means of the matrix Euclidean norm and prove a matrix Euclidean norm version of the Kantorovich inequality to limit upper or lower bounds of these efficiencies. It relaxes the assumptions that the covariance matrix is positive definite and the design matrix has full column rank.

MSC:62J05, 62H05, 62H20.

1 Introduction

Inequalities are studied and utilized widely in many fields such as in matrix theory, statistics and so on. In statistics, they are often used to make efficiency comparisons between two estimators. For example, Wang and Shao [1] have discussed the efficiency comparisons between the ordinary least squares estimator (OLSE) and the best linear unbiased estimator (BLUE) in linear models. In this paper, our goal is to make the comparison of efficiencies between OLSE and BLUE in a singular linear model by using matrix norm versions of the Kantorovich inequality involving a nonnegative definite matrix.

Consider the following linear regression model:

$$y=X\beta +\epsilon ,$$

(1.1)

where $y\in R^n$ is the vector of n observations, $X\in R^{n\times p}$ is the known design matrix, $\beta \in R^P$ is the unknown vector of regression coefficients and $\epsilon \in R^n$ is the error vector with mean vector zero and the covariance matrix Σ.

When X has full column rank and Σ is assumed to be positive definite, it is well known that the best linear unbiased estimator (BLUE) of β can be expressed as

$$\tilde{\beta }={\left(X^{\prime }{\mathrm{\Sigma }}^{-1}X\right)}^{-1}{X}^{\prime }{\mathrm{\Sigma }}^{-1}y$$

(1.2)

and the ordinary least squares estimator (OLSE) of β is given by

$$\hat{\beta }={\left(X^{\prime }X\right)}^{-1}{X}^{\prime }y.$$

(1.3)

According to the Löwner ordering, we can easily compute from (1.2) and (1.3) that $cov(\hat{\beta })-cov(\tilde{\beta })\ge 0$, which is nonnegative definite. Since there is no unique way to measure how ‘bad’ the OLSE can be with respect to the BLUE, various criteria have been considered in the literature; see, e.g., [2–6]. Among these criteria, the frequently used measure is the Watson efficiency [2] defined as follows:

$${\varphi }_1=\frac{|cov(\tilde{\beta })|}{|cov(\hat{\beta })|}=\frac{{|X^{\prime }X|}^2}{|X^{\prime }\mathrm{\Sigma }X|\cdot |X^{\prime }{\mathrm{\Sigma }}^{-1}X|},$$

(1.4)

where $|\cdot |$ indicates the determinant of the matrix concerned. The lower bound is provided by the Bloomfild-Watson-Knott inequality; see, e.g., [7, 8]. However, Yang and Wang [9] have shown that such a criterion is not always so satisfactory and provided an alternative form defined as the ratio of the Euclidean norms (or Frobenius norms) of the corresponding covariance matrices:

$${\varphi }_2=\frac{\parallel cov(\tilde{\beta })\parallel }{\parallel cov(\hat{\beta })\parallel }=\frac{\parallel {(X^{\prime }{\mathrm{\Sigma }}^{-1}X)}^{-1}\parallel }{\parallel {(X^{\prime }X)}^{-1}{X}^{\prime }\mathrm{\Sigma }X{(X^{\prime }X)}^{-1}\parallel }.$$

(1.5)

Many authors assume that the covariance matrix is nonsingular in their analysis of this classic linear model. But the number of characteristics that could be included in the model may be clearly limited by this assumption of nonsingularity. A few authors relax the condition of nonsingularity and consider a singular linear model. For example, Liski et al. [10] and Liu [4] make efficiency comparisons between the OLSE and BLUE in a singular linear model. In the present paper, the singular linear model is further studied.

The Watson efficiency ${\varphi }_1$ has been generalized to a weakly singular model; see, e.g., [11]. For a general case of the underlying singular linear model, it is not interesting because the denominator reduces to zero. In order to relax assumptions on the rank of X and Σ, we mainly discuss its alternative form based on the Euclidean norm [9, 12].

We hereinafter introduce some useful notations. Let the symbols $A^{\prime }$, $A^{-}$, $A^{+}$, $\mathcal{R}(A)$, $\mathcal{R}{(A)}^{\perp }$ and $rk(A)$ stand for the transpose, a generalized inverse, the Moore-Penrose inverse, the column space, the orthogonal complement of the column space and the rank of the matrix A, respectively. Moreover, write $P_A=A{A}^{+}=A{(A^{\prime }A)}^{+}{A}^{\prime }$ and $M_A=I-P_A$, in particular, $H=P_X$, $M=I-H$. ${\lambda }_i(A)$ denotes the i th largest eigenvalue of the matrix A.

2 A new Kantorovich-type inequality

We start with some lemmas which are very useful in the following.

Lemma 2.1 Let A be an $n\times n$ complex matrix and ${\lambda }_1,\dots ,{\lambda }_n$ be eigenvalues of A. Then we have

$$\sum _{i=1}^n|{\lambda }_i{|}^2\le {\parallel A\parallel }^2$$

and the equality holds if and only if A is a regular matrix.

Proof The proof is very easy, we therefore omit it here. □

Lemma 2.2 Let A be an $n\times n$ positive semidefinite Hermitian matrix and U be an orthogonal projection matrix with $rk(U)=k$. Then we have

$${\lambda }_{n-k+i}(A)\le {\lambda }_i(AU)\le {\lambda }_i(A),i=1,\dots ,k.$$

The proof can be found in [13]. See also [14].

Lemma 2.3 The Pólya and Szegö inequality

$$\left(\sum _{i=1}^n{a}_i^2\right)\left(\sum _{i=1}^n{b}_i^2\right)\le \frac{{(M_1{M}_2+m_1{m}_2)}^2}{4m_1{m}_2{M}_1{M}_2}{\left(\sum _{i=1}^n{a}_i{b}_i\right)}^2,$$

where $0<m_1\le a_i\le M_1$, $0<m_2\le b_i\le M_2$, $i=1,\dots ,n$.

Lemma 2.4 Let A and B be two $n\times n$ positive semidefinite Hermitian matrices, and U be an $n\times k$ matrix, $\mathcal{R}(A)\subset \mathcal{R}(B)$, $rk(B)=q$, $rk(BU)=t$, then

$${\lambda }_{q-t+i}\left(B^{-}A\right)\le {\lambda }_i\left({\left(U^{\ast }BU\right)}^{-}{U}^{\ast }AU\right)\le {\lambda }_i\left(B^{-}A\right),i=1,\dots ,t.$$

The proof can be found in [15]. See also [14].

Theorem 2.5 Let A be an $n\times n$ positive semidefinite Hermitian matrix and ${\lambda }_1\ge \cdots \ge {\lambda }_s>0$ ($s\le n$) be the ordered eigenvalues of A, and let U be an $n\times p$ complex matrix such that $U^{\ast }U=I_p$. If $p\le s$, we then have

Proof The proof is similar to Theorem 1 in [9], we therefore omit it here. □

3 The comparison of efficiencies

The Watson efficiency [2, 16] and its decompositions [11] are usually used to measure the efficiency of the ordinary least squares. However, Yang and Wang [9] show that such a criterion does not always work well in some cases and propose an alternative form

$$\rho =\frac{\parallel cov(X\tilde{\beta })\parallel }{\parallel cov(X\hat{\beta })\parallel }=\frac{\parallel X{(X^{\prime }{\mathrm{\Sigma }}^{-1}X)}^{-1}{X}^{\prime }\parallel }{\parallel X{(X^{\prime }X)}^{-1}{X}^{\prime }\mathrm{\Sigma }X{(X^{\prime }X)}^{-1}{X}^{\prime }\parallel }.$$

(3.1)

The above formula and its lower bound both require the covariance matrix Σ to be positive definite and the design matrix to have full column rank. This assumption limits clearly the number of characters which may be included in the model. We here generalize this formula to the situation where the matrices X and Σ can be of arbitrary rank.

In the following, we divide singular linear models into three categories in accordance with the assumptions on X and Σ. These categories are as follows:

• (1) $\mathcal{R}(X)\subset \mathcal{R}(\mathrm{\Sigma })$, $rk(X)=p$, Σ is possibly singular;

• (2) $\mathcal{R}(X)\subset \mathcal{R}(\mathrm{\Sigma })$, $rk(X)<p$, Σ is possibly singular;

• (3) Σ is possibly singular.

From now on, we always assume $rk(\mathrm{\Sigma })=s$ ($s<n$). Then any given singular linear model can be uniquely classified into i ($i=1,2,3$). Many authors have contributed to the theory in the literature; see, e.g., [17, 18]. The general representations for the BLUE of Xβ and their covariance matrices can be given respectively by

(3.2)

(3.3)

where $W=\mathrm{\Sigma }+XU{X}^{\prime }$ and here $U\ge 0$ is an arbitrary matrix such that $\mathcal{R}(W)=\mathcal{R}(X:\mathrm{\Sigma })$. In particular, ${\mathrm{\Sigma }}^{-}$ can play the same role as ${\mathrm{\Sigma }}^{-1}$ does when Σ is nonsingular as long as $\mathcal{R}(X)\subset \mathcal{R}(\mathrm{\Sigma })$. That is to say, $W^{-}$ can be replaced by ${\mathrm{\Sigma }}^{-}$ in this case. We then have

$$cov(X\tilde{\beta })=X{\left(X^{\prime }{\mathrm{\Sigma }}^{-}X\right)}^{-}{X}^{\prime }.$$

(3.4)

The covariance matrix of the well-known OLSE of Xβ is given by

$$cov(X\hat{\beta })=X{\left(X^{\prime }X\right)}^{-}{X}^{\prime }\mathrm{\Sigma }X{\left(X^{\prime }X\right)}^{-}{X}^{\prime }=H\mathrm{\Sigma }H.$$

(3.5)

In the following, we make efficiency comparisons between the OLSE and BLUE in a singular model according to the above category.

Firstly, we will discuss the category (1). The matrix product $X^{\prime }{\mathrm{\Sigma }}^{-}X$ is invariant for all the choices of generalized inverse ${\mathrm{\Sigma }}^{-}$ because of the column space inclusion $\mathcal{R}(X)\subset \mathcal{R}(\mathrm{\Sigma })$. Applying the rank rule of the matrix product [19], we can get that

$$rk\left(X^{\prime }\mathrm{\Sigma }X\right)=rk(\mathrm{\Sigma }X)=rk(X)-dim\mathcal{R}(X)\cap \mathcal{R}{(\mathrm{\Sigma })}^{\perp }=p.$$

(3.6)

Note that $\mathcal{R}(\mathrm{\Sigma })=\mathcal{R}({\mathrm{\Sigma }}^{+})$, and then we can conclude that $X^{\prime }\mathrm{\Sigma }X$ and $X^{\prime }{\mathrm{\Sigma }}^{+}X$ are both nonsingular. In the literature, such a model is often regarded as a weakly singular model or the Zyskind-Martin model [20]. In this model, the relative efficiency ρ becomes

$${\rho }_1=\frac{\parallel cov(X\tilde{\beta })\parallel }{\parallel cov(X\hat{\beta })\parallel }=\frac{\parallel X{(X^{\prime }{\mathrm{\Sigma }}^{+}X)}^{-1}{X}^{\prime }\parallel }{\parallel X{(X^{\prime }X)}^{-1}{X}^{\prime }\mathrm{\Sigma }X{(X^{\prime }X)}^{-1}{X}^{\prime }\parallel }.$$

(3.7)

It is easy to prove that ${\rho }_1\le 1$. The following theorem gives its lower bound.

Theorem 3.1 In the linear regression model (1.1), let ${\lambda }_1\ge \cdots \ge {\lambda }_s$ ($s<n$) be the ordered eigenvalues of Σ and X be an $n\times p$ design matrix with $rk(X)=p$, $\mathcal{R}(X)\subset \mathcal{R}(\mathrm{\Sigma })$. Then we have

$${\rho }_1\ge \frac{2p\sqrt{{\lambda }_1{\lambda }_p{\lambda }_{s-p+1}{\lambda }_s}}{({\lambda }_1{\lambda }_{s-p+1}+{\lambda }_s{\lambda }_p){\sum }_{i=1}^p\frac{{\lambda }_i}{{\lambda }_{s-p+i}}}.$$

Proof We can firstly compute that

There exists some orthogonal matrix P such that $\mathrm{\Sigma }=P\mathrm{\Lambda }{P}^{\prime }$, so , where $\mathrm{\Lambda }=diag({\lambda }_1,\dots ,{\lambda }_s,0,\dots ,0)$. Let , and then we have that and

Using Theorem 2.5, the result in Theorem 3.1 can be established. □

Secondly, we will consider the category (2). Let $rk(X)=r$ ($r<p$). Using equation (3.6), we can get that $rk(X^{\prime }\mathrm{\Sigma }X)=rk(X^{\prime }{\mathrm{\Sigma }}^{+}X)=r$. Analogically, the relative efficiency ρ becomes

$${\rho }_2=\frac{\parallel cov(X\tilde{\beta })\parallel }{\parallel cov(X\hat{\beta })\parallel }=\frac{\parallel X{(X^{\prime }{\mathrm{\Sigma }}^{+}X)}^{+}{X}^{\prime }\parallel }{\parallel X{(X^{\prime }X)}^{+}{X}^{\prime }\mathrm{\Sigma }X{(X^{\prime }X)}^{+}{X}^{\prime }\parallel }.$$

(3.8)

It is easy to prove that ${\rho }_2\le 1$. The following theorem gives its lower bound.

Theorem 3.2 In the linear regression model (1.1), let ${\lambda }_1\ge \cdots \ge {\lambda }_s$ ($s<n$) be the ordered eigenvalues of Σ and X be an $n\times p$ design matrix with $rk(X)=r$ ($r<p$), $\mathcal{R}(X)\subset \mathcal{R}(\mathrm{\Sigma })$. We then have

$${\rho }_2\ge \frac{2r\sqrt{{\lambda }_1{\lambda }_r{\lambda }_{s-r+1}{\lambda }_s}}{({\lambda }_1{\lambda }_{s-r+1}+{\lambda }_s{\lambda }_r){\sum }_{i=1}^r\frac{{\lambda }_i}{{\lambda }_{s-r+i}}}.$$

Proof It is easy to prove that

$$X{\left(X^{\prime }\mathrm{\Sigma }X\right)}^{+}{X}^{\prime }={\left(H{\mathrm{\Sigma }}^{+}H\right)}^{+}.$$

Then the proof is similar to Theorem 3.1, therefore we omit it here. □

Finally, we take into account the category (3). Owing to (3.3), we may write

$$cov(X\tilde{\beta })=cov(X\hat{\beta })-H\mathrm{\Sigma }M{(M\mathrm{\Sigma }M)}^{-}M\mathrm{\Sigma }H.$$

Therefore, we define

$${\rho }_3=\frac{\parallel cov(X\hat{\beta })-cov(X\tilde{\beta })\parallel }{\parallel cov(X\hat{\beta })\parallel }=\frac{\parallel H\mathrm{\Sigma }M{(M\mathrm{\Sigma }M)}^{-}M\mathrm{\Sigma }H\parallel }{\parallel H\mathrm{\Sigma }H\parallel }.$$

(3.9)

Let $dim\mathcal{R}(X)\cap \mathcal{R}(\mathrm{\Sigma })=g$ ($g\ge 0$) and $rk(X)=r$ ($r\le p$). Note that

$$\begin{array}{rcl}rk\left(X^{\prime }\mathrm{\Sigma }X\right)& =& rk\left(X^{\prime }X{\left(X^{\prime }X\right)}^{+}{X}^{\prime }\mathrm{\Sigma }X{\left(X^{\prime }X\right)}^{+}{X}^{\prime }X\right)\\ \le & rk\left(X{\left(X^{\prime }X\right)}^{+}{X}^{\prime }\mathrm{\Sigma }X{\left(X^{\prime }X\right)}^{+}{X}^{\prime }\right)\le rk\left(X^{\prime }\mathrm{\Sigma }X\right).\end{array}$$

Due to equation (3.6), we have

$$rk(H\mathrm{\Sigma }H)=rk(X)-dim\mathcal{R}(X)\cap \mathcal{R}{(\mathrm{\Sigma })}^{\perp }.$$

(3.10)

As a result from

$$\mathcal{R}(X)=\mathcal{R}(X)\cap (\mathcal{R}(\mathrm{\Sigma })\oplus \mathcal{R}{(\mathrm{\Sigma })}^{\perp })=(\mathcal{R}(X)\cap \mathcal{R}(\mathrm{\Sigma }))\oplus (\mathcal{R}(X)\cap \mathcal{R}{(\mathrm{\Sigma })}^{\perp }),$$

we then have

$$rk(H\mathrm{\Sigma }H)=r-(r-g)=g.$$

(3.11)

Similarly, we can obtain that

$$rk(M\mathrm{\Sigma }M)=rk(\mathrm{\Sigma }M)=rk(M)-dim\mathcal{R}(M)\cap \mathcal{R}{(\mathrm{\Sigma })}^{\perp }=dim\mathcal{R}(M)\cap \mathcal{R}(\mathrm{\Sigma }).$$

(3.12)

In view of $\mathcal{R}(M)=\mathcal{R}{(X)}^{\perp }$ and

$$\mathcal{R}(\mathrm{\Sigma })=\mathcal{R}(\mathrm{\Sigma })\cap (\mathcal{R}(X)\oplus \mathcal{R}{(X)}^{\perp })=(\mathcal{R}(\mathrm{\Sigma })\cap \mathcal{R}(X))\oplus (\mathcal{R}(\mathrm{\Sigma })\cap \mathcal{R}{(X)}^{\perp }),$$

we can get that

$$rk(M\mathrm{\Sigma }M)=rk(\mathrm{\Sigma }M)=s-g.$$

(3.13)

Theorem 3.3 In the linear regression model (1.1), let ${\lambda }_1\ge \cdots \ge {\lambda }_s$ ($s<n$) be the ordered eigenvalues of Σ and X be an $n\times p$ design matrix with $rk(X)=r$ ($r\le p$), $dim\mathcal{R}(X)\cap \mathcal{R}(\mathrm{\Sigma })=g$, $rk(H\mathrm{\Sigma }M{(M\mathrm{\Sigma }M)}^{-}M\mathrm{\Sigma }H)=h$, we then have

$${\rho }_3\le \{\begin{array}{cc}\frac{1}{2(s+r-n)}(\sqrt{\frac{{\lambda }_1{\lambda }_{n-r+1}}{{\lambda }_s{\lambda }_{s+r-n}}}+\sqrt{\frac{{\lambda }_s{\lambda }_{s+r-n}}{{\lambda }_1{\lambda }_{n-r+1}}})({\sum }_{i=1}^{s+r-n}\frac{{\lambda }_i}{{\lambda }_{n-r+i}}),\hfill & \mathit{\text{if }}h\le s+r-n,\hfill \\ \frac{1}{2(s+r-n)}(\sqrt{\frac{{\lambda }_1{\lambda }_{s-h+1}}{{\lambda }_s{\lambda }_h}}+\sqrt{\frac{{\lambda }_s{\lambda }_h}{{\lambda }_1{\lambda }_{s-h+1}}})({\sum }_{i=1}^h\frac{{\lambda }_i}{{\lambda }_{s-h+i}}),\hfill & \mathit{\text{if }}h>s+r-n.\hfill \end{array}$$

Proof For convenience, let $a=\parallel H\mathrm{\Sigma }M{(M\mathrm{\Sigma }M)}^{-}M\mathrm{\Sigma }H\parallel$ and $b=\parallel H\mathrm{\Sigma }H\parallel$. Then $H\mathrm{\Sigma }M{(M\mathrm{\Sigma }M)}^{-}M\mathrm{\Sigma }H$ is invariant for all the choices of generalized inverses ${(M\mathrm{\Sigma }M)}^{-}$. From Lemma 2.1, we can easily get that

$$a^2=\sum _{i=1}^h{\lambda }_i^2\left(H\mathrm{\Sigma }M{(M\mathrm{\Sigma }M)}^{-}M\mathrm{\Sigma }H\right)=\sum _{i=1}^h{\lambda }_i^2\left({(M\mathrm{\Sigma }M)}^{-}M\mathrm{\Sigma }H\mathrm{\Sigma }M\right).$$

Obviously, $h=rk(H\mathrm{\Sigma }M{(M\mathrm{\Sigma }M)}^{-}M\mathrm{\Sigma }H)\le rk(\mathrm{\Sigma }H)=g$ and $h\le rk(\mathrm{\Sigma }M)=s-g$. Since Σ and $\mathrm{\Sigma }H\mathrm{\Sigma }$ are positive semidefinite matrices and $\mathcal{R}(\mathrm{\Sigma }H\mathrm{\Sigma })\subset \mathcal{R}(\mathrm{\Sigma })$, we can derive from Lemma 2.4 that

$$a^2\le \sum _{i=1}^h{\lambda }_i^2\left({\mathrm{\Sigma }}^{-}\mathrm{\Sigma }H\mathrm{\Sigma }\right)=\sum _{i=1}^h{\lambda }_i^2\left(\mathrm{\Sigma }{\mathrm{\Sigma }}^{-}\mathrm{\Sigma }H\right)=\sum _{i=1}^h{\lambda }_i^2(\mathrm{\Sigma }H).$$

Here H is an orthogonal projection matrix, and then we obtain from Lemma 2.2 that

$$a^2\le \sum _{i=1}^h{\lambda }_i^2(\mathrm{\Sigma })\le \sum _{i=1}^h{\lambda }_i^2.$$

Furthermore, since $H\mathrm{\Sigma }H$ is a Hermitian matrix, by Lemma 2.1, we can get that

$$b^2=\sum _{i=1}^g{\lambda }_i^2(H\mathrm{\Sigma }H)=\sum _{i=1}^g{\lambda }_i^2(\mathrm{\Sigma }H).$$

The Sylvester theorem (see, e.g., [14]) shows that $n-r+g>s$. Analogically, we can get that

$$b^2\ge \sum _{i=1}^{s+r-n}{\lambda }_{n-r+i}^2.$$

Applying the well-known arithmetic-harmonic mean inequality, we have

$$\frac{1}{b^2}\le \frac{1}{{(s+r-n)}^2}\sum _{i=1}^{s+r-n}{\lambda }_{n-r+i}^{-2}.$$

Firstly, we suppose that $h\le s+r-n$. We can then compute that

$${\rho }_3^2=\frac{a^2}{b^2}\le \frac{1}{{(s+r-n)}^2}\sum _{i=1}^{s+r-n}{\lambda }_i^2\sum _{i=1}^{s+r-n}{\lambda }_{n-r+i}^{-2}.$$

By the Pólya and Szegö inequality and a nontrivial but elementary combinational argument, we can establish the first inequality. In fact, the second inequality is similar. □

4 Conclusions

In this article, we use several new matrix norm versions of the Kantorovich inequality involving a nonnegative definite matrix to make the comparison of efficiencies between OLSE and BLUE in a singular linear model. The singular linear model is divided into three categories in accordance with the assumptions on the ranks of X and Σ. We introduce some new relative efficiency criteria and their lower or upper bounds are given based on matrix norm inequalities in Theorem 3.1, Theorem 3.2 and Theorem 3.3.