Curve fitting

Conceptualization¶

Generally, we are given some datapoint pairs $x_i, y_i$. We suppose there is a function with parameters $\mathbf{a}$, $y(\mathbf{a};x)$ such that $y(\mathbf{a}, x_i)=y_i$. If $y(\mathbf{a};x)$ is linear in all the parameters (e.g. polynomials or RBFs), then we can just feed in $x_i$, $y_i$ pairs and construct a lienar system,

where each row of the system is determined by $y(\mathbf{a}, x_i) = y_i$.

NB: The function does not have to be linear in x! It is the parameters which we are solving for here.

If $\mathbf{A}$ is square, i.e. the number of parameters $a_i$ equals the number of pairs ${x_i,y_i}$ then this system can be solved exactly and we call this interpolation. Geometrically, this means our fit will pass through all of the data points.

Linear regression / linear least squares¶

In the general case, $\mathbf{A}$ is not square, e.g. we have more data points than parameters, and we want to find the best fit for $\mathbf{a}$; i.e. the parameters $\mathbf{a}$ that minimizes

The set of residuals $r_i$ is the vector $\mathbf{r}$ and our goal is to minimize $\| \mathbf{r}\|$. This is called linear regression / linear least squares, and we already know how to do this generally through the pseudoinverse,

Recall that the pseudoinverse recovers the exact inverse for square matricies (an exact fit is the best fit by definition!)

Foreshadowing nonlinear least squares¶

In general, curve fitting is an optimization problem since $y(a,x)$ can be nonlinear in $a$ (e.g. $y = a e^{b x}$). This is nonlinear least squares which is an optimization problem and will be covered later. Technically, nonlinear least squares is the true generalization which includes linear regression and our methods will also be applicable. However, linear regression is common and robust enough that it is maintained as a separate tool.