I.3.4.2: Nonlinear Least Squares

Consider the problem of minimizing $$\sum_{i=1}^n (y_i-\sum_{j=1}^m \phi_j(x_i,\theta)\beta_j)^2,$$ with the $\phi_j$ known nonlinear functions.

Again parameters separate naturally into two blocks $\beta$ and $\theta,$ and finding the optimal $\beta$ for given $\theta$ is again linear regression.

The best way of finding the optimal $\theta$ for given $\beta$ will typically depend on a more precise analysis of the problem, but one obvious alternative is to linearize the $\phi_j$ and apply Gauss-Newton.