Linear Mixed Model | Block Relaxation Algorithms in Statistics -- Part I

I.6.5.5: Linear Mixed Model

This example is taken from a paper of De Leeuw and Liu \cite{dlliu}, which describes the algorithm in detail. We simply give a list of results that show augmentation at work. We maximize a multinormal likelihood, not a least squares criterium.

Result: If $A=B+TCT',$ with $B,C>0$ , $y'A^{-1}y=\min_x (y-Tx)'B^{-1}(y-Tx)+x'C^{-1}x.$

Result: If $A=B+TCT',$ with $B,C>0$ , $\log\det{A}=\log\det{B}+\log\det{C}+\log\det{C^{-1}+T'B^{-1}T}.$

Result: If $A=B+TCT'$ then $\begin{multline*} \log\det{A}+y'A^{-1}y=\min_{x}\ \log\det{B}+\log\det{C}+\\ +\log\det{C^{-1}+T'B^{-1}T}+\\ +(y-Tx)'B^{-1}(y-Tx)+x'C^{-1}x. \end{multline*}$

Result: If $T>0$ then $\log\det{T}=\min_{S>0}\log\det{S}+\hbox{ tr }S^{-1}T-p,$ with the unique minimum attained at $S=T.$

We can use these four results to augment the original maximum likelihood problem. $\begin{multline*} \log\det{A}+y'A^{-1}y=\min_{x,S>0}\ \log\det{B}+\log\det{C}+\\ +\log\det{S}+\hbox{ tr }S^{-1}(C^{-1}+T'B^{-1}T)+\\ +(y-Tx)'B^{-1}(y-Tx)+x'C^{-1}x. \end{multline*}$ Minimize over $x,S,B,C$ using block-relaxation. The conditional minimizers are $\begin{align*} S&=C^{-1}+T'B^{-1}T,\notag\\ C&=S^{-1}+xx',\notag\\ B&=TS^{-1}T'+(y-Tx)(y-Tx)',\notag\\ x&=(T'B^{-1}T+C^{-1})^{-1}T'B^{-1}y.\notag \end{align*}$