I.3.7.4: Multinormal Maximum Likelihood

The negative log-likelihood for a multinormal random sample is $$f(\theta,\xi)=n\log\mathbf{det}(\Sigma(\theta))+\sum_{i=1}^n (x_i-\mu(\xi))'\Sigma^{-1}(\theta)(x_i-\mu(\xi)).$$ The vector of means $\mu$ depends on the parameters $\xi$ and the matrix of covariances $\Sigma$ depends on $\theta$. We assume the two sets of parameters are separated, in the sense that they do not overlap.

Oberhofer and Kmenta [1974] study this case in detail and give a proof of convergence, which is actually the expected special case of Zangwill's theorem.

Suppose we have a normal GLM of the form $$\underline{y}\sim\mathcal{N}[X\beta,\sum_{s=1}^p\theta_s\Sigma_s]$$ where the $\Sigma_s$ are known symmetric matrices. We have to estimate both $\beta$ and $\theta,$ perhaps under the constraint that $\sum_{s=1}^p\theta_s\Sigma_s$ is positive semi-definite.

This can be done, in many case, by block relaxation. Finding the optimal $\beta$ for given $\theta$ is just weighted linear regression. Finding the optimal $\theta$ for given $\beta$ is more complicated, but the problem has been studied in detail by Anderson and others.

For further reference, we give the derivatives of the log-likelihood function for this problem. $$\frac{\partial\mathcal{L}}{\partial\theta_s}= \text{tr }\Sigma^{-1}\Sigma_s -\text{tr }\Sigma^{-1}\Sigma_s\Sigma^{-1}S.$$ $$\frac{\partial^2\mathcal{L}}{\partial\theta_s\partial\theta_t}= \text{tr }\Sigma^{-1}\Sigma_s\Sigma^{-1}\Sigma_t\Sigma^{-1}S+ \text{tr }\Sigma^{-1}\Sigma_t\Sigma^{-1}\Sigma_s\Sigma^{-1}S- \text{tr }\Sigma^{-1}\Sigma_s\Sigma^{-1}\Sigma_t.$$ Taking expected values in Equation \ref{E:And2} gives $$\mathbf{E}\left\{\frac{\partial^2\mathcal{L}}{\partial\theta_s\partial\theta_t}\right\}= \text{tr }\Sigma^{-1}\Sigma_s\Sigma^{-1}\Sigma_t.$$