I.3.6.4: Block Newton Methods

In block Newton methods we update using $$y_s=x_s-[\mathcal{D}_{ss}f(y_1,\cdots,y_{s-1},x_s,\dots,x_p)]^{-1}\mathcal{D}_{s}f(y_1,\cdots,y_{s-1},x_s,\dots,x_p)$$ It follows that at a point $x$ where the derivatives vanish we have $$\mathcal{D}_tF_s(x)=\delta^{st}I-[\mathcal{D}_{ss}f(x)]^{-1}\mathcal{D}_{st}f(x)$$ and in particular $\mathcal{D}_sF_s(x)=0$. The iteration matrix $\mathcal{M}(x)$ is the same as the one in section block optimization methods or, and consequently the convergence rate is the same as well. We will again have the same rate if we make more than one Newton step in each block update.

Of course the single-step block Newton method does not guarantee decrease of the loss function, and consequently needs to be safeguarded in some way.