I.6.5.3: Least Squares Factor Analysis

In LS factor analysis we want to minimize $$\sigma(A)=\sum_{i=1}^m\sum_{j=1}^m w_{ij}(r_{ij}-\sum_{s=1}^p a_{is}a_{js})^2,$$ with $$w_{ij}= \begin{cases} 0,& \text{if $i=j$,}\\ 1,& \text{if $i\not=j$.} \end{cases}$$ We augment by adding the communalities, i.e. the diagonal elements of $R$ as variables, and by using ALS over $A$ and the communalities. For a complete $R,$ minimizing over $A$ just means computing the $p$ dominant eigenvalues-eigenvectors. This algorithm dates back to the thirties, when it was proposed by Thomson and others.

Think of this as an algorithm for updating communalities. We have $$H^{(k+1)}=\mathbf{diag}(I-R_p^{(k)})$$ where $R_p^{(k)}$ is the best rank p approximation to $R-H^{(k)}$.