I.3.7.1: Canonical Correlation

Canonical Correlation is a matrix problem in which the notion of blocks of variables is especially natural. The problem can be formulated in various ways, but we prefer a least squares formulation. We want to minimize $\sigma(A,B)\mathop{=}\limits^{\Delta}\mathbf{tr }\ (XA-ZB)'(XA-ZB)$ over $A$ and $B.$ In order to avoid boring complications which merely lead to more elaborate notations we again suppose that $X'X=I$ and $Z'Z=I$.

Note that this problem is basically a multivariate version of the block least squares problem with $Y=0$ in the previous example. There are some crucial differences, however. The fact that $Y=0$ means that $A=B=0$ trivially minimizes $\sigma$. Thus we need to impose some normalization condition such as $A'A=I$ and/or $B'B=I$ to exclude this trivial solution. Nevertheless, in our analysis we shall initially proceed without actually using normalization.

Start with $A^{(0)}$. To find the optimal $B^{(k)}$ for given $A^{(k)}$ we compute $R'A^{(k)}=B^{(k)},$ and then we update $A$ with $RB^{(k)}=A^{(k+1)},$ where $R\mathop{=}\limits^{\Delta}X'Z$, as before. Thus $A^{(k+1)}=R'RA^{(k)}$ and $A^{(k)}=(R'R)^kA^{(0)}$. Clearly $A^{(k)}\rightarrow 0$ if $R'R\lesssim I$, which implies convergence to the correct, but trivial, solution $A=B=0$.

Suppose $R'R=K\Lambda K'$ is the eigen-decomposition and define $\Delta^{(k)}\mathop{=}\limits^{\Delta}K'A^{(k)}$, so that $\Delta^{(k)}=\Lambda^k\Delta^{(0)}.$ As in the previous example $$\frac{\|\Delta^{(k)}\|}{\lambda_+^k}\rightarrow\|\tilde\Delta^{(0)}\|,$$ where $\tilde\Delta^{(0)}$ consists of the columns corresponding with the dominant eigenvalue. Again $$\frac{\|\Delta^{(k+1)}\|}{\|\Delta^{(k)}\|}\rightarrow\lambda_+.$$

Now consider $\Xi^{(k)}=A^{(k)}((A^{(k)})'A^{(k)})^{-\frac12}=\Lambda^k\Delta_0(\Delta_0'\Lambda^{2k}\Delta_0)^{-\frac12}$