All about the ⊥ with its applications in the linear statistical models

© 2015 Augustyn Markiewicz and Simo Puntanen licensee De Gruyter Open.

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 License.

Open Mathematics Open Access

Review Article

Augustyn Markiewicz and Simo Puntanen*

All about the ? with its applications in the linear statistical models

Abstract:For annmreal matrixAthe matrixA^?is defined as a matrix spanning the orthocomplement of the column space ofA, when the orthogonality is defined with respect to the standard inner producthx;yi D x⁰y. In this paper we collect together various properties of the?operation and its applications in linear statistical models.

Results covering the more general inner products are also considered. We also provide a rather extensive list of references.

Keywords:Best linear unbiased estimator, Column space, Generalized inverse, Linear statistical model, Orthocom- plement, Orthogonal projector

MSC:15A99, 62H12, 62J05

DOI 10.1515/math-2015-0005

Received December 13, 2013; accepted April 14, 2014.

1 Introduction

Consider a columnwise partitioned matrixA D .a1 W : : : W am/ 2 R^nm (the set ofnm matrices with real elements). Then the column space ofAis defined as

C.A/D fz2RⁿWzDAtDa1t1C Camtmfor somet2R^mg:

The notation_C.A/^? refers to the orthocomplement of_C.A/, i.e., the set of vectors which are orthogonal (with respect to the standard inner product) to every vector of_C.A/:

C.A/^?D fu2RⁿWu⁰AtD0for allt2R^mg: Thus, because

C.A/^?D fu2RⁿWu⁰AtD0for allt2R^mg D fu2RⁿWA⁰uD0g; we have

C.A/^?DN.A⁰/Dthe null space ofA⁰:

NowA^?is defined as a matrix whose column space is_C.A^?/DC.A/^?DN.A⁰/. In view of decomposition RⁿDC.A/˚C.A/^?;where˚refers to the direct sum, the rank ofA^?is rank.A^?/Dn rank.A/:The set of all matricesA^?is denoted asfA^?gand hence:

Z2 fA^?g ” (a)A⁰ZD0 and (b) rank.Z/Dn rank.A/: (1)

Augustyn Markiewicz:Department of Mathematical and Statistical Methods, Pozna´n University of Life Sciences, Wojska Polskiego 28, PL-60637 Pozna´n, Poland, E-mail: amark@au.poznan.pl

*Corresponding Author: Simo Puntanen:School of Information Sciences, FI-33014 University of Tampere, Finland, E-mail: simo.puntanen@uta.fi

We immediately observe thatZ2 fA^?g ” A 2 fZ^?g:TriviallyA^?is unique only whenAis a nonsingular square matrix in which caseA^?D0. Notice that

A2R^nm H) A^?2R^ns; where sn rank.A/:

In this paper our purpose is to review various features of the?operation, the “perp-operation”, say, and in particular, to present several useful applications related to linear statistical models. Results covering the more general inner products are also considered. We believe that our review provides a useful summary of the?operation and thereby increases the insights and appreciation of this, seemingly simple, operation.

2 A

in terms of generalised inverses

The generalised inverses offer a very handy tool for explicit expressions of theA^?, and in this section we give a short tour into such possibilities. MatrixG2R^mnis a generalised inverse ofA2R^nmif

AGADA; (mp1)

and it is the Moore–Penrose inverse, denoted asA^C, if it also satisfies the following three conditions:

(mp2)GAGDG; (mp3).AG/⁰DAG; (mp4).GA/⁰DGA:

If Gsatisfies the conditionAGA D A, we may denoteG D A , orG 2 fA g. As the excellent references for the generalised inverses, see Ben-Israel & Greville [10] and Rao & Mitra [46]. In particular, for more of about the Moore (of Moore & Penrose), see Ben-Israel [11].

It is well known that the nullspaceN.A/can be expressed as N.Anm/DC.Im A A/ ;

whereA can be any generalized inverse ofA. Hence we can express_C.A/^?in terms ofA :

C.A/^?DN.A⁰/DCŒIn .A⁰/ A⁰DCŒIn .A /⁰A⁰ : (2) The last equality above follows from the fact

f.A /⁰g D f.A⁰/ g: (3) Notice that it is a bit questionable to write .A /⁰ D .A⁰/ because (3) means the equality between twosets.

However, for the (unique) Moore–Penrose inverse we always have.A^C/⁰ D .A⁰/^C:In light of (2), we have, for example, the following choices forA^?(recalling thatA2R^nm):

In .A⁰/ A⁰; In .A /⁰A⁰; In A.A⁰A/ A⁰; (4) where we have used the factA.A⁰A/ 2 f.A⁰/ g. By replacing.A⁰/ with.A⁰/^Cin (4) and using

.A⁰/^CD.A^C/⁰; .A⁰/^CA⁰DAA^C;

we get

In AA^CWDIn PAWDQA2 fA^?g: (5) It can be shown that ifGsatisfies the conditions (mp1) and (mp3), i.e.,G 2 fA₁₃gthenAGis unique and therebyAA₁₃DAA^C;and henceIn AA₁₃is one choice forA^?.

The notationsPA andQA in (5) refer to the orthogonal projectors onto_C.A/ (with respect to the standard inner product) and_C.A/^?, respectively. MatrixPis defined as the orthogonal projector onto_C.A/if it satisfies the following conditions:

PDP⁰DP² and _C.P/DC.A/; (6)

which can be shown to be equivalent toP.A W A^?/ D .A W 0/;where.A WA^?/and.A W 0/denote partitioned matrices.

The matrixPsatisfying (6) is unique and can be written asPADAA^CDAA₁₃DA.A⁰A/ A⁰, where the last expression is invariant for any choice of.A⁰A/ ; this follows from Rao & Mitra [46, Lemma 2.2.4], which says that for nonnullAandC, the matrix productAB Cis invariant with respect to the choice of the generalized inverseB if and only if_C.C/C.B/and_C.A⁰/C.B⁰/. Notice thatAA is not necessarily an orthogonal projector: it is idempotent and it satisfies_C.AA /DC.A/but it is not necessarily symmetric.

Below is a summary of some of the expressions forA^?with obvious extensions to.A⁰/^?in terms of generalised inverses.

Theorem 2.1. ForA2R^nm, denoteQADIn AA^CDIn PA. Then (a) In .A⁰/ A⁰2 fA^?g;

(b) In .A /⁰A⁰2 fA^?g; (c) In PADQA2 fA^?g; (d) In AA₁₃2 fA^?g; (e) Im A A2 f.A⁰/^?g;

(f) Im A⁰.AA⁰/ ADIm A^CADIm PA⁰DQA⁰2 f.A⁰/^?g:

Obviously the orthogonal projectorQADIn AA^Cis often a convenient choice forA^?because it is symmetric and idempotent.

3 Some specific formulas

Suppose thatZis a choice forA^?. Then, for a comformable matrixB, we have

ZB2 fA^?g (7)

whenever rank.ZB/Drank.Z/. According to Marsaglia & Styan [34, Cor. 6.2] (see Theorem 4.1 below), rank.ZB/Drank.Z/ dim_C.Z⁰/\C.B/^?;

and thereby (7) holds if and only if_C.Z⁰/\C.B/^?D f0g:Thus we have the following simple result:

Theorem 3.1. LetA2R^nm, andB2R^nq. Then for anyA^?2Rⁿⁿwe have A^?B2 fA^?g ” CŒ.A^?/⁰\C.B/^?D f0g: In particular, choosingQAasA^?yields

QAB2 fA^?g ” C.AWB/DRⁿ; where.AWB/denotes the partionedn.mCq/matrix.

In the next theorem we take a look at the perps of some particular partitioned matrices.

Theorem 3.2. LetA2R^nm, andB2R^nq. Then for anyA^?we have

(a) A^? 0 0 Iq

! 2

8

<

: Anm

0qm

!?9

=

;

;

(b) In

B⁰

! A^?2

8

<

:

AnmBnq

0 Iq

!?9

=

;

;

(c) In

B⁰

! 2

8

<

: B_nq

Iq

!_?9

=

; :

Proof. Part (a) is obvious as the orthogonality condition corresponding to (1a) trivially holds and rank A^? 0

0 Iq

!

Dn rank.A/Cq:

To prove (b), we observe that

Œ.A^?/⁰W .A^?/⁰B A B 0 Iq

! D0:

Moreover, the rank of ^In_B0

A^?is rank

.A^?/⁰W .A^?/⁰B

DrankŒ.A^?/⁰Dn rank.A/ ; while

rank AnmBnq

0 Iq

!?

DnCq rank Anm

0

!

rank Bnq

Iq

!

DnCq rank.A/ qDn rank.A/ : Thus (b) is confirmed. Part (c) can be proved in the corresponding way.

Theorem 3.3. ConsiderA2R^nmandB2R^tm. Then for anyA^?andB^?we have

A^? 0 0 B^?

! 2

8

<

: A B

!?9

=

; if and only if _C.A⁰/\C.B⁰/D f0g:

Proof. The orthogonality condition (1a) obviously holds while rank A^? 0

0 B^?

!

DnCt rank.A/ rank.B/;

rank A B

!?

DnCt rank.A/ rank.B/Cdim_C.A⁰/\C.B⁰/:

Thus the proof is completed.

Remark 3.4. It might be a bit tempting to rewrite part (a) of Theorem 3.2 as

Anm

0qm

!?

D A^? 0 0 Iq

!

: (8)

However, expression like(8)is obviously problematic, and the meaning of the above notation should be clarified.

One interpretation for(8)might be to agree that it means that 8

<

: Anm

0qm

!?9

=

; D

( A^? 0 0 Iq

!)

: (9)

In other words, the sets of matrices are identical. However, the statement(9)is incorrect as can be concluded by Theorem 3.5 below.

Let us ask the following: which matricesB2R^npandD2R^qpsatisfy the following:

A^?B_np 0 Dqp

! 2

8

<

: A_nm 0qm

!_?9

=

;

‹

We first observe that equation

.A^?/⁰ 0 B⁰ D⁰

! Anm

0qm

!

D 0nm

0qm

!

holds if and only if_C.B/ C.A/^?. Supposing that_C.B/ C.A/^?, then, in view of Marsaglia & Styan [34, Cor. 19.1], the rank of ^A₀^?_D^B

is additive on the Schur complement, i.e., rank A^? B

0 D

!

Drank.D/Crank.A^? BD 0/Drank.D/Crank.A^?/Drank.D/Cn rank.A/ : On the other hand, because

rank Anm

0qm

!?

DnCq rank.A/ ; we immediately obtain the following:

Theorem 3.5. ConsiderA2R^nm,B2R^np, andD2R^qp. Then the relation

A^? B 0 D

! 2

8

<

: A 0

!_?9

=

; holds if and only if_C.B/C.A/^?andDhas full row rank.

4 Two rank formulas and a decomposition of orthogonal projector

Two particular rank formulas in terms of the orthocomplement are worth special praising due to their numerous applications particularly when dealing with linear statistical models: the rank of the productAnaBamand the rank of the partitioned matrix.AnaWBnb/.

Theorem 4.1. The rank of the partitioned matrix.AnaWBnb/can be expressed as

rank.AWB/Drank.A/CrankŒ.A^?/⁰BDrank.A/Crank.B⁰A^?/ ; (10) and the rank of the matrix productAnaBamis

rank.AB/Drank.A/ dim_C.A⁰/\C.B/^?: (11)

In terms of an arbitrary generalized inverseA , (10) can be expressed as

rank.AWB/DrankŒAW.In AA /BDrank.A/CrankŒ.In AA /BDrank.A/CrankŒ.In PA/B : (12) As a reference to (10) and (12) we may mention Marsaglia & Styan [34, Th. 5]. For the references to (11), see, e.g., Marsaglia & Styan [34, Cor. 6.2], Rao [43, p. 28], and Zyskind & Martin [59, p. 1194]. We may also mention that O.M. Baksalary & Trenkler [8] provide several expressions for the ranks of a product of two matrices and of a column-wise partitioned matrix as well as an extensive list of related references.

Several applications of (10) and (11) appear in Puntanen, Styan & Isotalo [39, Ch. 5]. One example concerns the decomposition of the column space_C.XWVX^?/, whereX2R^npandVis annn(symmetric) nonnegative definite matrix. Such a situation occurs when we consider the general linear model

yDXˇC"; denoted as _M D fy;Xˇ;Vg; (13) whereXis a knownnpmodel matrix, the vectoryis an observablen-dimensional random vector,ˇis ap1 vector of unknown parameters, and"is an unobservable vector of random errors with expectation E."/ D 0;and covariance matrix cov."/DV. Then we have the following; see, e.g., Rao [45, Lemma 2.1].

Theorem 4.2. Consider the linear model_MD fy;Xˇ;Vg, defined as in(13). Then C.XWV/DC.XWVX^?/DC.X/˚C.VX^?/ :

Moreover, if the model is correct, in which case it is called consistent, then the observed (realized) value of the random vectorysatisfies

y2C.XWV/ : (14)

For a discussion concerning the consistency concept, see, e.g., Puntanen & Styan [38], J.K. Baksalary, Rao &

Markiewicz [5], Groß [16, p. 314], and Tian et al. [53]. In this paper, we assume that the corresponding consis- tency holds whatever model we have.

When working with linear models, we often need to consider the orthogonal projector onto the column space of the partitioned matrix. Then the following theorem appears to be very convenient in various connections; see, e.g., Puntanen, Styan & Isotalo [39, Th. 8] and Seber & Lee [50, Appendix B].

Theorem 4.3. The orthogonal projector (with respect to the standard inner product) onto the column space C.AnaWBnb/can be decomposed as

P_.AWB/DPACP.In PA/BDPACP_C.AWB/\C.A/?: (15) We complete this section by some remarks on the explicit expression for the intersection of_C.A/and_C.B/. For a reference, see Rao & Mitra [46, Complement 7, p. 118].

Theorem 4.4. Consider the matricesAnaandBnband denoteQBDIn PB. Then C.A/\C.B/DC

A.A⁰B^?/^? DC

A.A⁰QB/^? DC

A.Ia PA⁰QB/

DC AŒIa .A⁰QBA/ A⁰QBA

: It is obvious that

C.A/\C.B/^?DC

A.A⁰B/^? DC

A.Ia PA⁰B/ : In particular, ifX2R^npandV_nnis nonnegative definite, then

C.X/\C.V/^?DC

X.X⁰V/^? DC

X.X⁰VX/^?

DC XŒIp .X⁰VX/ X⁰VX

; and in view of_C.M/\C.V/^?DC.XWV/^?,

MŒIn .MVM/ MVM2˚

.XWV/^? ;

whereMDIn PX. Notice also that according to Theorem 4.3 we haveP_.XWV/DPXCPMVand thereby In P_.XWV/DM PMVDM.In PMV/2˚

.XWV/^? :

5 Orthocomplement when the inner product matrix is V

5.1

V

Consider now the inner product inRⁿdefined ashx;yi_VDx⁰Vy;whereVis a positive definite symmetric matrix.

The orthocomplement ofC.Anm/with respect to this inner product is

C.A/^?_V D fy2Rⁿ Wz⁰A⁰VyD0for allz2R^mg:

ByA^?_V we will denote any matrix whose column space is_C.A/^?_V. Recall thatA^?_I is shortly denoted asA^?. We have

C.A/^?_V D fy2RⁿWA⁰VyD0g DN.A⁰V/DC.VA/^?DC.V ¹A^?/ ;

where the last equality can be concluded from

A⁰VV ¹A^?D0 H) C.V ¹A^?/N.A⁰V/ ; and

rank.V ¹A^?/Drank.A^?/Dn rank.A/Ddim_C.VA/^?: Notice that corresponding to (1),

Z2 fA^?_Vg ” (a)A⁰VZD0and (b) rank.Z/Dn rank.A/:

Remark 5.1. Obviously we can writeV ¹A^?2 fA^?_VgandV ¹A^?2 f.VA/^?g. Question: Is it correct to write

fA^?Vg D f.VA/^?g‹

It is easy to confirm that the answer is positive.

Now we have the following decomposition:

RⁿDC.A/˚C.A/^?_V DC.A/˚C.V ¹A^?/ ; and hence everyy2Rⁿhas a unique representation as a sum

yDAbCV ¹A^?cDyC Py;

for somebandc. The vectory DAbis the orthogonal projection ofyonto_C.A/along_C.A/^?_V. The orthogonal projectorP_AIVis such a matrix which transformsyinto its projectiony, i.e.,P_AIVyDyDAb. Its explicit unique representation is

P_AIVDA.A⁰VA/ A⁰V:

We may mention that part (a) of Theorem 3.2 holds even if the inner product matrix isV, i.e., A^?_V 0

0 Iq

! 2

8

<

: A_nm

0qm

!_?

V

9

=

; :

Similarly Theorems 3.3 and 3.5 hold also when all orthocomplements are taken with respect to the inner product matrixV.

5.2

V

LetVbe a singular nonnegative definite matrix. Thenht;ui_VDt⁰Vuis a semi-inner product and the corresponding seminorm (squared) isktk²_VDt⁰Vt. For a singular nonnegative definite matrixVwe can define the matrixA^?_V again as any matrix spanningC.A/^?_V, and so

C.A^?_V/DC.A/^?_V DN.A⁰V/DC.VA/^?:

As noted by Puntanen, Styan & Isotalo [39, §2.5] for (even) a singularVwe do have the decomposition

RⁿDC.A/CC.A^?_V/DC.A/CC.VA/^?; (16) but, however, the above decomposition is not necessarily a direct sum. For any nonnegative definiteVwe have, on account of Theorem 4.1,

dim_C.VA/^?Dn rank.VA/DŒn rank.A/Cdim_C.A/\C.V/^?; which means that (16) becomes a direct sum decomposition if and only if_C.A/\C.V/^?D f0g.

For the characterization of the generalized orthogonal projector, see Mitra & Rao [36]. Some related considera- tions appear also in Harville [19, §14.12.i], Rao & Rao [47, p. 81], and Tian & Takane [54, 55].

5.3 Some further considerations

Consider the linear model_MD fy;Xˇ;Vg, defined as in (13), and letVbe positive definite. Then we have observed that the following sets are identical:

(a) _C.X/^?_V 1; (b) _C.VX^?/ ; (c) _N.X⁰V ¹/ ; (d) _C.V ¹X/^?; (e) _N.P_XIV ¹/ ; (f) _C.In P_XIV ¹/ :

For (a), . . . , (f) above, see also Puntanen, Styan & Isotalo [39, §5.13]. WhenVis singular, the above considerations become more complicated. A very convenient tool appears to be the following class of matrices:

WD fW2RⁿⁿWWDVCXUX⁰; _C.W/DC.XWV/g: (17) In (17)Ucan be anyppmatrix as long as_C.W/D C.X WV/is satisfied. Of course,Ucan be chosen as0if C.X/C.V/which happens, for example, whenVis positive definite. The setWof matrices has an important role in the theory of linear models. Below are listed some useful equivalent statements concerningW:

C.X/C.W/ ; (18a)

C.XWV/DC.W/ ; (18b)

X⁰W Xis invariant for any choice ofW ; (18c)

C.X⁰W X/DC.X⁰/for any choice ofW ; (18d)

X.X⁰W X/ X⁰W XDXfor any choices of the generalized inverses involved. (18e) Moreover, each of these statements is equivalent to_C.X/ C.W⁰/, and hence to the statements (18b’)–(18e’) obtained from (18b)–(18e), by settingW⁰in place ofW. As the references to (18), we may mention J.K. Baksalary, Puntanen & Styan [4, Th. 2], J.K. Baksalary & Mathew [3, Th. 2], and Harville [19, p. 468].

According to Puntanen, Styan & Isotalo [39, §5.13] the following now holds.

Theorem 5.2. Suppose thatXis annpmatrix,Vis annnnonnegative definite matrix andW2^W, whereW is defined as in(17). Then

C.VX^?/DC.W XWIn W W/^?;

whereW is an arbitrary(but fixed)generalized inverse ofW. The column space_C.VX^?/can be expressed also as

C.VX^?/DC

.W /⁰XWIn .W /⁰W⁰?

: Moreover, letVbe possibly singular and assume that_C.X/C.V/. Then

C.VX^?/DC.V XWIn V V/^?C.V X/^?; where the inclusion becomes equality if and only ifVis positive definite.

Remark 5.3. It is of interest to note that the perp symbol?drops down, so to say, very “nicely” whenVis positive definite:

C.VX^?/^?DC.V ¹X/ ;

but whenVis singular we have to use a much more complicated rule to drop down the?symbol:

C.VX^?/^?DC.W XWIn W W/ ; whereW2^W.

Remark 5.4. Let us next prove the following: IfW2^W, whereWis defined as in(17), then

C.VX^?/DC.W X/^? ” C.XWV/DRⁿ: (19)

We first observe that

C.VX^?/DC.W XWIn W W/^?DC.W X/^?\C.In W W/^?:

Thus we always have_C.VX^?/C.W X/^?;where the equality appears only ifdim_C.W X/^? Drank.VX^?/.

Now we have

rank.VX^?/Drank.W/ rank.X/ ;

dim_C.W X/^?Dn rank.W X/Dn rank.X/ ; from which our claim(19)follows.

For completeness we state the following related result, due to Rao & Mitra [46, p. 140].

Theorem 5.5. Consider the linear model_M D fy;Xˇ;Vgand denoteWDVCXUX⁰, where_C.W/DC.XWV/, and letW be an arbitrary generalized inverse ofW. Then

C.W X/˚C.X/^?DRⁿ; _C.W X/^?˚C.X/DRⁿ; CŒ.W /⁰X˚C.X/^?DRⁿ; _CŒ.W /⁰X^?˚C.X/DRⁿ:

6 Statistical examples

6.1 Centering

We would like to start with a simple but at the same time very important orthocomplement in statistics: the set of vectors orthogonal to the vector of ones, that is,C.1n/^?, where1n D .1; 1; : : : ; 1/⁰ 2 Rⁿ. In what follows, we most of the time drop off the subscript from the vector1n; from the context its dimension should be obvious. The orthogonal projector ontoC.1n/isP₁D ¹_n11⁰WDJand the orthogonal projector ontoC.1n/^?isIn 1

n11⁰WDC;

Cis the centering matrix.

Consider then2data matrixUpartitioned as

UD.xWy/D 0 B B B B

@ x1y1

x2y2

::: ::: xnyn

1 C C C C A

D 0 B B B B

@ u⁰_.1/

u⁰_.2/

::: u⁰_.n/

1 C C C C A :

Hereu.i / D ^xyⁱ_i

2 R² represents theith case or theith observation in the observation space, and the vectors x;y2Rⁿrepresent the two variables in the variable space. LetuN D ^xy^N_N

2R²denote the mean vector ofx- and y-variables andSthe sample covariance matrix:

N

uD ¹_nU⁰1nD _n¹.u.1/Cu.2/C Cu.n//D xN N y

!

;

SD _n¹₁U⁰CUD _n¹₁

n

X

iD1

.u.i / u/.uN .i / u/N ⁰:

Now the following theorem is easy to confirm; for details, see, e.g., Puntanen, Styan & Isotalo [39, Ch. 3].

Theorem 6.1. For conformable matrices, the following statements hold:

(a) The vectoryNND Ny1is the orthogonal projection of the variable vectoryonto the column space_C.1/:yNND Ny1D JyDP1y.

(b) The centered variable vectoryQis the orthogonal projection ofyonto the column space_C.1/^?:yQ Dy JyD CyD.In P1/y:

(c) Let the variances of the variablesxandybe nonzero, i.e.,x…C.1/andy…C.1/. Then the sample correlation coefficientrxyis the cosine of the angle between the centered variable vectors:

rxyDcos.Cx;Cy/Dcos.x;Q y/Q D x⁰Cy px⁰Cxy⁰Cy:

(d) yis centered ” y2C.1/^?DC.C/DN.1⁰/.

Next we shortly consider a typicalnpmodel matrixXpartitioned asXD.1Wx1W: : :Wxk/D.1WX0/, and so pDkC1. The sample covariance matrix of thex-variables isSxxD _n¹₁X⁰₀CX0and the sample correlation matrix isRxx D Œdiag.S_xx/ ¹⁼²SxxŒdiag.S_xx/ ¹⁼²: While calculating the correlations, we assume that allx-variables have nonzero variances, that is, the matrix diag.Txx/is positive definite, or in other words:xi …C.1/; iD1; : : : ; k:

Theorem 4.1 implies then the following result:

Theorem 6.2. The rank of the model matrixXD.1WX0/can be expressed as

rank.X/D1Crank.X0/ dimC.1/\C.X0/Drank.1WCX0/D1Crank.CX0/D1Crank.S_xx/ ; and thereby

rank.Sxx/Drank.X/ 1Drank.CX0/Drank.X0/ dim_C.1/\C.X0/ :

If allx-variables have nonzero variances, i.e., the correlation matrixRxx is properly defined, thenrank.R_xx/ D rank.Sxx/. Moreover, the following statements are equivalent:

(a) det.Sxx/¤0; (b) rank.X/DkC1; (c) rank.X0/Dkand1…C.X0/:

For the rank of of the sample covariance matrix, see Trenkler [56]. As regards the geometry and linear models, the reader may take a look at Margolis [31], Herr [22], and Seber [49].

6.2 Estimability in a simple ANOVA

Following Puntanen, Styan & Isotalo [39, §1.2], consider a simple analysis-of-variance (ANOVA) model

AW yD 0 B B B B

@

1n1 1n1 0 : : : 0 1n2 0 1n2 : : : 0 ::: ::: ::: : :: ::: 1ng 0 0 : : :1ng

1 C C C C A

0 B B B B B B

@ 1

2

::: g

1 C C C C C C A

C"DXˇC"D.1nWX0/

! C";

wherenDn1C Cng. As the rank of then.gC1/model matrixXisgwe know thatˇis not estimable under_A. Which parametric functions ofˇare estimable?

We recall thatK⁰ˇis estimable if it has an unbiased linear estimator, say Aywith property E.Ay/ D AXˇ DK⁰ˇfor allˇ2R^p, i.e.,AXDK⁰. Hence the parametric functionk⁰ˇis estimable under_A if and only if

k2C.X⁰/DC 1⁰_n X⁰₀

!

DC 1⁰_g Ig

!

: (20)

In view of part (c) of Theorem 3.2, one choice for ₁₀

g

Ig

?

is 1

1g

, i.e., 1

1g

! WDu2

8

<

: 1⁰_g Ig

!?9

=

; :

Hence, according to (20), the parametric functionk⁰ˇis estimable if and only if k2 C 1⁰_g

Ig

!?!?

DC.u/^?;

i.e.,

k⁰uD0; where uD 1 1g

!

: (21)

We can also study the estimability of a parametric function of1; : : : ; g(dropping off the parameter); denote this function as`⁰. Then

.0; `⁰/

! D`⁰;

and on account of (21), the estimability condition for`<sup>0</sup>becomes`⁰1gD0.

6.3 Best linear unbiased estimator, BLUE

An unbiased linear estimatorGyforXˇis defined to be the best linear unbiased estimator, BLUE, forXˇunder the model_M D fy;Xˇ;Vgif cov.Gy/Lcov.Ly/for allLWLXDX;where “L” refers to the Löwner partial order- ing. In other words,Gyhas the smallest covariance matrix in the Löwner sense among all linear unbiased estimators.

The following theorem gives the “fundamental BLUE equation”; see, e.g., Rao [40], Zyskind [58], J.K. Baksalary [1], and O.M. Baksalary & Trenkler [6, 7].

Theorem 6.3. Consider the general linear model_MD fy;Xˇ;Vg, defined as in(13). Then the estimatorGyis the BLUEforXˇif and only ifGsatisfies the equation

G.XWVX^?/D.XW0/ : (22) Notice also that even thoughGin (22) may not be unique, the numerical observed value of Gyis unique (with probability1) once the random vectoryhas obtained its value in the space_C.X W VX^?/. The set of matricesG satisfying (22) is sometimes denoted asfP_XjVX?g.

Remark 6.4. At this point we may take a liberty to make a short side trip to the notation P_AjB in the spirit of Rao [45] and Kala [25]. Supposing that_C.A/and_C.B/are (virtually) disjoint, theny2C.AWB/has a unique representation as a sumy D yA CyB, where yA 2 C.A/,yB 2 C.B/. A matrixPwhich transforms every y 2 C.A W B/into its projectionyA is called a projector onto_C.A/along _C.B/. It appears that the projector PWDP_AjBonto_C.A/along_C.B/may be defined by the equation

P_AjB.AWB/D.AW0/ :

Kala [25, Lemma 2.5] proved that if_C.A/\C.B/D f0g DC.C/\C.D/, then

fP_CjDg fP_AjBg ” C.A/C.C/ and _C.B/C.D/ : Moreover, Rao [45] showed that

.P_VAjA?CP_A?jVA/zDz; .P_VA?jACP_AjVA?/yDy; P_AjVA?yD.In P⁰_A?IV/y;

hold for allz2C.A^?WVA/DC.A^?WV/andy2C.AWVA^?/DC.AWV/.

We shall use the short notation

HDPX; MDIn H;

and thereby the ordinary least squares estimator.OLSE/ofXˇisHy; we will denoteHyD Xˇ, whereO ˇOis any solution toX⁰XˇDX⁰y. IfXhas full column rank thenˇis estimable and its OLSE isˇOD.X⁰X/ ¹X⁰yDX^Cy.

Characterizing the equality of the OLSE and the BLUE of Xˇ has received a lot of attention in the statistical literature, the major breakthroughs being made by Rao [40], Zyskind [58], and Kruskal [27]; for a review, see Puntanen & Styan [37], and for some special remarks, Markiewicz, Puntanen & Styan [33], and O.M. Baksalary, Trenkler & Liski [9].

Theorem 6.3 gives immediately several equivalent characterizations for the OLSE and the BLUE to be equal, some of them are collected in Theorem 6.5. Notice that then the equality between OLSE and BLUE occurs with probability1but in what follows, we drop off the phrase “with probability1”.

Theorem 6.5. Consider the general linear model_M D fy;Xˇ;Vg. ThenOLSE.Xˇ/DBLUE.Xˇ/if and only if any one of the following five equivalent conditions holds:

(a) HVDVH, (b)HVMD0, (c)_C.VX/C.X/,

(d) _C.X/has a basis comprising a set ofrDrank.X/orthonormal eigenvectors ofV,

(e) VDaInCHN1HCMN2Mfor somea2R, and matricesN1andN2such thatVis nonnegative definite.

Using, for example, Rao & Mitra [46, p. 24] and Ben-Israel & Greville [10, p. 52], we obtain the following.

Theorem 6.6. The general solution forGsatisfyingG.XWVX^?/D.XW0/can be expressed, for example, in the following ways:

(a) G1D.XW0/.XWVX^?/ CF1QW, (b) G2DX.X⁰W X/ X⁰W CF2QW,

(c) G3DIn VX^?Œ.X^?/⁰VX^? .X^?/⁰CF3QW, (d) G4DH HVX^?Œ.X^?/⁰VX^? .X^?/⁰CF4QW,

whereF1: : : ;F4are arbitrary matrices,QWDIn PW, andW2^W, whereWis defined as in(17).

In view of the consistency condition (14), we havey2C.W/and hence the termsFiQWydisappear with probabi- lity1. We observe, for example, that

BLUE.Xˇ/DHy HVM.MVM/ MyDOLSE.Xˇ/ HVM.MVM/ My;

or, denoting shortlyXˇQDBLUE.Xˇ/andXˇODOLSE.Xˇ/,

XˇO XˇQDHVM.MVM/ My: It is easy to confirm that

cov.Xˇ/Q DHVH HVM.MVM/ MVHDcov.Xˇ/O HVM.MVM/ MVH: (23)

WhenXhas full column rank andVis positive definite, thenˇOD.X⁰X/ ¹X⁰yandˇQD.X⁰V ¹X/ ¹X⁰V ¹y while the corresponding covariance matrices are

cov.ˇ/O D.X⁰X/ ¹X⁰VX.X⁰X/ ¹; cov.ˇ/Q D.X⁰V ¹X/ ¹: (24) On the other hand, in light of (23) we have

cov.ˇ/Q Dcov.ˇ/O .X⁰X/ ¹X⁰VM.MVM/ MVX.X⁰X/ ¹: (25) It is interesting to note that in (25) the covariance matrixVneed not be positive definite. IfVis positive definite, then combining (24) and (25) yields the following:

Theorem 6.7. Consider the linear model_M D fy;Xˇ;Vg, whereXhas full column rank andVis positive definite.

Then

cov.ˇ/Q D.X⁰V ¹X/ ¹D.X⁰X/ ¹ŒX⁰VX X⁰VM.MVM/ MVX.X⁰X/ ¹

Dcov.ˇ/O X^CVM.MVM/ MV.X^C/⁰; (26) and

cov.Xˇ/Q DX.X⁰V ¹X/ ¹X⁰DHVH HVM.MVM/ MVH:

Among the first places where (26) occurs are the papers by Khatri [26, Lemma 1], Rao [40, Lemmas 2a, 2b, 2c], and Rao [43, p. 77]. Theorem 6.7 offers a convenient way to express the so-called Watson efficiency, see Watson [57, p. 330], as

D jcov.ˇ/Q j

jcov.ˇ/O j D jX⁰VX X⁰VM.MVM/ MVXj jX⁰Xj ²

jX⁰VXj jX⁰Xj ² D jX⁰VX X⁰VM.MVM/ MVXj jX⁰VXj

D jIp .X⁰VX/ ¹X⁰VM.MVM/ MVXj:

Abovejjrefers to the determinant. For related considerations, see Puntanen, Styan & Isotalo [39, §10.7–10.8] and the references therein.

In this context we may briefly say a couple of words about the matrix product MP WDM.MVM/ M;

which appears in several formulas above. IfVis positive definite andV¹⁼²is its positive definite symmetric square root, andZis a matrix having full column rank with the property_C.Z/DC.M/, then we obviously have

MP DM.MVM/ MDV ¹⁼²P_V1=2ZV ¹⁼²DZ.Z⁰VZ/ ¹Z⁰;

which is clearly unique. In general, the matrixMP is not necessarily unique with respect to the choice of.MVM/ . Moreover, for positive definiteVwe have

MP DM.MVM/ MD.MVM/^CDV ¹ V ¹X.X⁰V ¹X/ X⁰V ¹;

and ifHPVMD0then, see Isotalo, Puntanen & Styan [24, Th. 2.1],

PVMPP VDPVM.MVM/ MPVDV^C V^CX.X⁰V^CX/ X⁰V^C:

The matrixMP is very handy in many connections related to linear model_M D fy;Xˇ;Vg. For example, the ordinary, unweighted sum of squares of errors SSE is defined as

SSEDSSE.I/Dmin

ˇ ky Xˇk²Dy⁰My; while the weighted SSE is (whenVis positive definite)

SSE.V/Dmin

ˇ ky Xˇk²_V 1 D ky P_XIV 1yk²_V 1 Dy⁰ŒV ¹ V ¹X.X⁰V ¹X/ X⁰V ¹y Dy⁰M.MVM/ MyDy⁰MyP :

In the general case, the weighted SSE can be defined as

SSE.V/D.y Xˇ/Q ⁰W .y Xˇ/ ;Q whereWDVCXUX⁰, withC.W/DC.XWV/. Then, again,

SSE.V/D.y Xˇ/Q ⁰W .y Xˇ/Q Dy⁰MyP :

For further properties ofM, see Puntanen, Styan & Isotalo [39, Ch. 15] and Isotalo, Puntanen & Styan [24]. SomeP related considerations appear also in Markiewicz [32, pp. 415–416], LaMotte [28, pp. 323–324], and Searle, Casella

& McCulloch [48, pp. 451–452].

What about if we require thateveryrepresentation of the BLUE under_M1 D fy;Xˇ;V1gcontinues to be BLUE under_M2D fy;Xˇ;V2g? The answer is given in Theorem 6.8. For the proof and related discussion, see, e.g., J.K. Baksalary & Mathew [2, Th. 3], Mitra & Moore [35, Th. 4.1–4.2], Rao [41, Lemma 5], Rao [42, Th. 5.2, Th. 5.5], Rao [44, p. 289], Tian [52], Tian & Takane [54, 55], and Hauke, Markiewicz & Puntanen [21].

Theorem 6.8. Consider the linear models_M1 D fy;Xˇ;V1gand_M2D fy;Xˇ;V2g. Then every representation of theBLUEforXˇunder_M1remains theBLUEforXˇunder_M2if and only if any of the following equivalent conditions hold:

(a) _C.V2X^?/C.V1X^?/,

(b) V2DaV1CXN1X⁰CV1X^?N2.X^?/⁰V1, for somea2R, and matricesN1andN2such thatV2is nonne- gative definite.

6.4 The reduced model

Let us consider the partitioned linear model M12 D fy;X1ˇ₁CX2ˇ₂;Ing, where X D .X1 W X2/ has full column rank,X1 2 R^np1,X2 2 R^np2,p D p1Cp2. In light of the projector decomposition (15), we have HDP.X1WX2/DPX1CPM1X2;whereM1DIn PX1and thereby

HyDX1ˇO1CX2ˇO2DPX1yCPM1X2y: (27) Premultiplying (27) byM1gives

M1X2ˇO₂DPM1X2yDM1X2.X⁰₂M1X2/ ¹X⁰₂M1y: (28) In view of (11), rank.M1X2/D rank.X2/ D p2, and hence the left-mostM1X2can be cancelled from (28) and thus we obtain

ˇO₂D.X⁰₂M1X2/ ¹X⁰₂M1yWDˇO₂._M12/: (29) Premultiplying the modelM12by the orthogonal projectorM1yields the reduced model

M¹²1D fM1y;M1X2ˇ2;M1g:

Taking a look at the models, we can immediately make an important conclusion: the OLS estimators ofˇ2under the models_M12and_M121coincide:

ˇO₂._M12/DˇO₂._M121/D.X⁰₂M1X2/ ¹X⁰₂M1y: (30) The equality (30) is the result that Davidson & MacKinnon [14, §2.4] call the Frisch–Waugh–Lovell theorem; see Frisch & Waugh [15], Lovell [29, 30].

Let us take a quick look at the more general case when the partitioned linear model is_M12 D fy;X1ˇ1C X2ˇ₂;Vg. PremultiplyingM¹²by the orthogonal projectorM1yields the reduced model

M121D fM1y;M1X2ˇ₂;M1VM1g: What about the BLUE ofM1X2ˇ₂in the reduced modelM121? Let us denote

fBLUE.M1X2ˇ2jM¹²/g D fAyWAyis BLUE forM1X2ˇ2g:

Before proceeding we notice thatK⁰₂ˇ2is estimable under_M12 if and only if there exists a matrixLsuch that L.X1WX2/D.0WK2/, i.e., see Groß & Puntanen [17, Lemma 1],

C.K2/C.X⁰₂X^?₁/DC.X⁰₂M1/:

Moreover, it is easy to confirm thatK⁰₂ˇ2is estimable under_M12if and only ifK⁰₂ˇ2is estimable under_M121. Then we can formulate the generalized Frisch–Waugh–Lovell theorem as follows; see, e.g., Groß & Puntanen [17, Th. 4].

Theorem 6.9. Every representation of theBLUEofM1X2ˇ2under_M12 D fy;X1ˇ1CX2ˇ2;Vgremains the BLUEunder_M121D fM1y;M1X2ˇ₂;M1VM1gand vice versa, i.e., the sets of theBLUEs coincide:

fBLUE.M1X2ˇ₂jM12/g D fBLUE.M1X2ˇ₂jM121/g:

In other words: LetK⁰₂ˇ₂be an arbitrary estimable parametric function under_M12. Then every representation of theBLUEofK⁰₂ˇ₂under_M12remains theBLUEunder_M121and vice versa.

LetX D .X1 W X2/have full column rank, and_C.X/ C.V/, but Vis possibly singular. Then it appears that corresponding to (29) we have

ˇQ₂._M12/D.X⁰₂MP1X2/ ¹X⁰₂MP1y; whereMP1DM1.M1VM1/ M1DV ¹ V ¹X1.X⁰₁V ¹X1/ X⁰₁V ¹.

For further references related to the Frisch–Waugh–Lovell theorem, see for example, Bhimasankaram & Sen- gupta [12, Th. 6.1], Sengupta & Jammalamadaka [51, §7.10], and Groß & Puntanen [18].

6.5 Best linear unbiased predictor, BLUP

Letyf denote aq1unobservable random vector containing new future observations. The new observations are assumed to follow the linear modelyf DXfˇC"f, whereXf is a knownqpmatrix,ˇis the same vector of unknown parameters as in_M D fy;Xˇ;Vg, and"f is aq-dimensional random error vector associated with new observations. Then

E y

yf

!

D Xˇ Xfˇ

! D X

Xf

!

ˇ; cov y yf

!

D V V12

V21V22

! : For brevity, we denote

Mf D (

y yf

!

; X

Xf

!

ˇ; V V12

V21 V22

!)

: (31)

The linear predictorByis said to be unbiased foryf if E.yf By/ D 0for allˇ 2 R^p. This is equivalent to BXDXf. Now a linear unbiased predictorByis the best linear unbiased predictor, BLUP, foryf, if the Löwner ordering cov.yf By/Lcov.yf Fy/holds for allFsuch thatFyis an unbiased linear predictor foryf.

The following theorem characterizes the BLUP; see, e.g., Christensen [13, p. 294], and Isotalo & Puntanen [23, p. 1015].

Theorem 6.10. Consider the linear model_Mf (with new unobserved future observations), defined as in(31), where C.X_f⁰ / C.X⁰/. The linear predictorByis the best linear unbiased predictor(BLUP)foryf if and only ifB satisfies the equation

B.XWVX^?/D.Xf WV21X^?/:

The linear mixed model_L, say, can be specified as

yDXˇCZC"; i.e., L D fy;XˇCZ;D;Rg; (32) whereˇis a vector of fixed parameters anda vector of random ones, with the known covariance matrices cov."/D Rand cov./DD, and expectations E."/D0, E./D0. We assume that the random effectand error term"are uncorrelated and thereby cov.y/DZDZ⁰CRD˙, say. Taking as the “new observation” it is easy to conclude, in view of Theorem 6.10, that the following holds.

Theorem 6.11. Consider the linear mixed model_L, defined as in(32). The the linear predictorAyis theBLUP ofunder the mixed model_L if and only if

A.XW˙X^?/D.0WDZ⁰X^?/ :

6.6 Stochastic restrictions

Let us consider the fixed effects partitioned model

FWyDXˇCZC"; cov.y/Dcov."/DR;

where bothˇ and are fixed (but unknown) coefficients, and supplement _F with the stochastic restrictions y0DC"0, where cov."0/DD:This supplement can be expressed as the partitioned model:

FD fy;X;Vg D (

y y0

!

; X Z

0 Iq

! ˇ

!

; R 0

0 D

!) :

We will need the matrix X^? for which, according to part (b) of Theorem 3.2, one choice is ^In_Z0

M; where MDIn P_X, and so we have

VX^? D R 0 0 D

! In

Z⁰

!

MD RM DZ⁰M

!

: (33)

Now the estimatorByis the BLUE forXunder the model_Fif and only ifBsatisfies the equation

B.XWVX^?/D.XW0/ : (34) Substituting (33) into (34) yields

B11B12

B21B22

!

X Z RM 0 Iq DZ⁰M

!

D X Z 0 0 Iq 0

!

: (35)

Using (35) Haslett & Puntanen [20, Th. 1] show that all properties of BLUEs and BLUPs in mixed model_L can be considered using the augmented model_F, where bothˇandare fixed parameters. Using the connection between the mixed model_L the augmented model_F, the following result follows from Theorem 6.8 immediately.

Theorem 6.12. Consider two mixed models:_Li D fy;XˇCZ;Di;Rig;and denote˙i DZDiZ⁰CRi and Vi D ^R_{0 D}^{i 0}_i

; i D 1; 2:Then every representation of theBLUEforXˇunder_L1remains theBLUEforXˇ under_L2and every representation of theBLUPfor under_L1remains theBLUPfor under_L2if and only if any of the following equivalent conditions holds:

(a) Every representation of theBLUEforXunder_F1remains theBLUEforXunder_F2. (b) _C.V2X^?/C.V1X^?/.

(c) _C R2M D2Z⁰M

!

C R1M D1Z⁰M

! :

(d) _C ˙2M D2Z⁰M

!

C ˙1M D1Z⁰M

! : (e) The matrixV₂can be expressed as

V2DaV1CXN1X⁰CV1X^?N2.X^?/⁰V1

for somea2Rand matricesN1andN2such thatV2is nonnegative definite.

Acknowledgement: Thanks go to the referees for helpful remarks. Part of this research was done during the meet- ing of a Research Group on Mixed and Multivariate Models in the Mathematical Research and Conference Center, Be¸dlewo, Poland, October 2013, supported by the Stefan Banach International Mathematical Center.

References

[1] Baksalary J.K., An elementary development of the equation characterizing best linear unbiased estimators, Linear Algebra Appl., 2004, 388, 3–6

[2] Baksalary J.K., Mathew T., Linear sufficiency and completeness in an incorrectly specified general Gauss–Markov model, Sankhy ¯a A, 1986, 48, 169–180

[3] Baksalary J.K., Mathew T., Rank invariance criterion and its application to the unified theory of least squares, Linear Algebra Appl., 1990, 127, 393–401

[4] Baksalary J.K., Puntanen S., Styan G.P.H., A property of the dispersion matrix of the best linear unbiased estimator in the general Gauss–Markov model, Sankhy ¯a A, 1990, 52, 279–296

[5] Baksalary J.K., Rao C.R., Markiewicz A., A study of the influence of the “natural restrictions” on estimation problems in the singu- lar Gauss–Markov model, J. Statist. Plann. Inference, 1992, 31, 335–351

[6] Baksalary O.M., Trenkler G., A projector oriented approach to the best linear unbiased estimator, Statist. Papers, 2009, 50, 721–733

[7] Baksalary O.M., Trenkler G., Between OLSE and BLUE, Aust. N. Z. J. Stat., 2011, 53, 289–303

[8] Baksalary O.M., Trenkler G., Rank formulae from the perspective of orthogonal projectors, Linear Multilinear Algebra, 2011, 59, 607–625

[9] Baksalary O.M., Trenkler G., Liski E.P., Let us do the twist again. Statist. Papers, 2013, 54, 1109–1119 [10] Ben-Israel A., Greville T.N.E., Generalized inverses: theory and applications,2nd Ed., Springer, New York, 2003 [11] Ben-Israel A., The Moore of the Moore–Penrose inverse, Electron. J. Linear Algebra, 9, 150–157, 2002

[12] Bhimasankaram P., Sengupta D., The linear zero functions approach to linear models, Sankhy ¯a B, 1996, 58, 338–351