Singular Values and Singular Vectors

Suppose $A_{12}$ is an $n_1\times n_2$ matrix, $A_{11}$ is an $n_1\times n_1$ symmetric matrix, and $A_{22}$ is an $n_2\times n_2$ symmetric matrix. Define $$f(x_1,x_2)=\frac{x_1'A_{12}x_2}{\sqrt{x_1'A_{11}x_1}\sqrt{x_2'A_{22}x_2}}.$$ Consider the problem of finding the maximum, the minimum, and other stationary values of $f$.

In order to make the problem well-defined and interesting we suppose that the symmetric partitioned matrix $$A\mathop{=}\limits^{\Delta}=\begin{bmatrix}A_{11}&A_{12}\\A_{21}&A_{22}\end{bmatrix}$$ is positive semi-definite. This has some desirable consequences.

Proposition: Suppose the symmetric partitioned matrix $$A\mathop{=}\limits^{\Delta}=\begin{bmatrix}A_{11}&A_{12}\\A_{21}&A_{22}\end{bmatrix}$$ is positive semi-definite. Then

• both $A_{11}$ and $A_{22}$ are positive semi-definite,
• for all $x_1$ with $A_{11}x_1=0$ we have $A_{21}x_1=0$,
• for all $x_2$ with $A_{22}x_2=0$ we have $A_{12}x_2=0$.

Proof: The first assertion is trivial. To prove the last two, consider the convex quadratic form $$q(x_2)=x_1'A_{11}x_1+2x_1'A_{12}x_2+x_2'A_{22}x_2$$ as a function of $x_2$ for fixed $x_1$. It is bounded below by zero, and thus attains its minimum. At this minimum, which is attained at some $\hat x_2$, the derivative vanishes and we have $A_{22}\hat x_2=-A_{21}x_1$ and thus $q(\hat x_2)=x_1'A_{11}x_1-\hat x_2'A_{22}\hat x_2.$ If $A_{11}x_1=0$ then $q(\hat x_2)\leq 0$. But $q(\hat x_2)\geq 0$ because the quadratic form is positive semi-definite. Thus if $A_{11}x_1=0$ we must have $q(\hat x_2)=0$, which is true if and only if $A_{22}\hat x_2=-A_{21}x_1=0$. QED

Now suppose $$A_{11}=\begin{bmatrix}K_1&\overline{K}_1\end{bmatrix} \begin{bmatrix}\Lambda_1^2&0\\0&0\end{bmatrix} \begin{bmatrix}K_1'\\\overline{K}_1'\end{bmatrix},$$ and $$A_{22}=\begin{bmatrix}K_2&\overline{K}_2\end{bmatrix} \begin{bmatrix}\Lambda_2^2&0\\0&0\end{bmatrix} \begin{bmatrix}K_2'\\\overline{K}_2'\end{bmatrix}$$ are the eigen-decompositions of $A_{11}$ and $A_{22}$. The $r_1\times r_1$ matrix $\Lambda_1^2$ and the $r_2\times r_2$ matrix $\Lambda_2^2$ have positive diagonal elements, and $r_1$ and $r_2$ are the ranks of $A_{11}$ and $A_{22}$.

Define new variables $$\begin{align*} x_1&=K_1\Lambda_1^{-1}u_1+\overline{K}_1v_1,\tag{1a}\\ x_2&=K_2\Lambda_2^{-1}u_2+\overline{K}_2v_2.\tag{1b} \end{align*}$$ Then $$f(x_1,x_2)=\frac{u_1'\Lambda_1^{-1}K_1'A_{12}K_2\Lambda_2^{-1}u_2}{\sqrt{u_1'u_1}\sqrt{u_2'u_2}},$$ which does not depend on $v_1$ and $v_2$ at all. Thus we can just consider $f$ as a function of $u_1$ and $u_2$, study its stationary values, and then translate back to $x_1$ and $x_2$ using $(1a)$ and $(1b)$, choosing $v_1$ and $v_2$ completely arbitrary.

Define $H\mathop{=}\limits^{\Delta}\Lambda_1^{-1}K_1'A_{12}K_2\Lambda_2^{-1}$. The stationary equations we have to solve are $$\begin{align*} Hu_2&=\rho u_1,\\ H'u_1&=\rho u_2, \end{align*}$$ where $\rho$ is a Lagrange multiplier, and we identify $u_1$ and $u_2$ by $u_1'u_1=u_2'u_2=1$. It follows that $$\begin{align*} HH'u_1&=\rho^2 u_1,\\ H'Hu_2&=\rho^2 u_2, \end{align*}$$ and also $\rho=f(u_1,u_2)$.