Non-Linear Regression

A straight line of the wrong shape

Five plots are given 1, 3, 5, 7 and 9 units of water, and the crops are 3.8, 6.2, 7, 6.2 and 3.8. Too little water and too much both give a small crop, so the points rise to a peak and fall back.

The least-squares line through them is level, y = 5.4, the mean crop. It lies above the two outer points and below the middle three, and its residuals are −1.6, 0.8, 1.6, 0.8 and −1.6. Their squares total 2.56 + 0.64 + 2.56 + 0.64 + 2.56 = 8.96. The line does no better than the mean, so $R^2=0$R² = 0.

The best straight line, y = 5.4. The two outer points lie below it and the middle three above it, a pattern no straight line can remove.

The same rule, over a curve

Least squares does not need a straight line. For a quadratic model $y=a{x}^2+bx+c$y = ax² + bx + c, it chooses a, b and c to make the total of the squared vertical gaps as small as possible, exactly as it chose the gradient and intercept of a line.

For these points the result is $y=-0.2x^2+2x+2$y = −0.2x² + 2x + 2, which is $y=7-0.2{\left(x-5\right)}^2$y = 7 − 0.2(x − 5)². At x = 1 it gives −0.2 + 2 + 2 = 3.8, at x = 3 it gives −1.8 + 6 + 2 = 6.2, and at x = 5 it gives 7. These points lie exactly on the parabola, so every residual is 0 and the squared gaps total 0, against 8.96 for the line.

The least-squares quadratic, $y=7-0.2{\left(x-5\right)}^2$y = 7 − 0.2(x − 5)², passes through all five points, so no gap is left.

Five families of curves

A calculator offers five families besides the line, each with its own shape. A quadratic, $a{x}^2+bx+c$ax² + bx + c, has one turning point. A cubic, $a{x}^3+b{x}^2+$ax³ + bx² + cx + d, can turn twice. An exponential, $a{b}^x$ab^x, multiplies by the same factor b at every step of 1 in x, so it climbs or falls ever faster and never turns back. A power model, $a{x}^b$ax^b, such as $3x^2$3x² or $5\sqrt{x}$5√x, passes through the origin when b is positive. A sine model, a sin(bx + c) + d, rises and falls in the same cycle again and again.

The data’s shape, and what is known about the situation, decide the family. Least squares then decides the numbers in it. Monthly rainfall that rises and falls in the same cycle every year calls for a sine model; a cubic turns at most twice and could not repeat the second year.

Counts that double

A count of colonies is 0.5 thousand at week 0, then 1, 2, 4 and 8 thousand at weeks 2, 4, 6 and 8. It doubles every two weeks, so it is multiplied by the same factor over every equal step: the exponential family, $y=a{b}^x$y = ab^x.

Here $y=0.5\times 2^{\frac{x}{2}}$y = 0.5 × 2^(x/2). Since $2^{\frac{x}{2}}={\left(\sqrt{2}\right)}^x$2^(x/2) = (√2)^x, that is $y=a{b}^x$y = ab^x with a = 0.5 and $b=\sqrt{2}$b = √2, about 1.414: the count grows by 41.4 percent each week.

The curve $y=0.5\times 2^{\frac{x}{2}}$y = 0.5 × 2^(x/2) through the five counts. Each step of two weeks doubles the count, so the curve gets steeper and steeper.

Straightening an exponential with logarithms

Take logarithms to base 10 of both sides of $y=a{b}^x$y = ab^x: log y = log a + x log b. That is a straight line, with log y up the side and x, unchanged, across. Its gradient is log b and its intercept is log a. Only y is logged.

For the colonies, log y at weeks 0, 2, 4, 6 and 8 is −0.301, 0, 0.301, 0.602 and 0.903. It rises by 0.301 every two weeks, so the points lie on a line with gradient 0.301 ÷ 2 = 0.1505 and intercept −0.301.

Read the model back: log b = 0.1505, so $b=10^{0.1505}=1.414$b = 10^0.1505 = 1.414, and log a = −0.301, so $a=10^{-0.301}=0.5$a = 10^(−0.301) = 0.5. The gradient is not b and the intercept is not a; each is the logarithm of one.

The same five counts with log y plotted against the week. The points lie on a straight line with gradient 0.1505 and intercept −0.301.

A power model needs both logged

For a power model $y=a{x}^n$y = ax^n, logarithms give log y = log a + n log x. Now the line has log x across and log y up the side, so both variables are logged, and the gradient is n itself.

Take x = 1, 2, 4, 8 and y = 3, 12, 48, 192. The values of log x are 0, 0.301, 0.602 and 0.903, and of log y 0.477, 1.079, 1.681 and 2.283. The gradient is (2.283 − 0.477) ÷ 0.903 = 2, so n = 2, and the intercept is 0.477, so $a=10^{0.477}=3$a = 10^0.477 = 3. The model is $y=3x^2$y = 3x².

A line fitted to the logged values makes the squared gaps in log y smallest, not the squared gaps in y. The two fits are usually close, and they agree exactly when the points lie on the curve, as these do.

Transform back before predicting

To predict the colonies at week 10, use the straight line first: log y = −0.301 + 0.1505 × 10 = 1.204. That is the logarithm of the count, not the count. Then $y=10^{1.204}=16$y = 10^1.204 = 16 thousand, which is 8 thousand doubled, as two more weeks should give.

Week 10 lies beyond the data, which run to week 8, so 16 thousand assumes the doubling carries on.

The rule never changes

For the arch, the line leaves squared gaps totaling 8.96 and the quadratic leaves 0. Least squares chose the best member of each family; neither number says which family to choose. That choice comes from the shape of the data and from what the quantities are.

A family that contains another, as the quadratics contain every straight line (a = 0), always leaves smaller or equal squared gaps on the same data, so a smaller total is not enough on its own to prefer it. A quadratic is right for the arch because the crop rises to a peak and falls, not only because it fits better.

The usual mistakes

Fitting a straight line to data that curve. The residuals then follow a pattern, here negative at both ends and positive in the middle.

Reading the gradient of log y against x as b. A gradient of 0.1505 means $b=10^{0.1505}=1.414$b = 10^0.1505 = 1.414.

Logging x as well as y for an exponential model, or only y for a power model. $a{b}^x$ab^x needs log y against x; $a{x}^n$ax^n needs log y against log x.

Giving the value on the straight line as the prediction. log y = 1.204 is a logarithm; the count is $10^{1.204}=16$10^1.204 = 16.

Bacteria counted every hour

In the application below, a count N of bacteria is modeled by $N=a{b}^x$N = ab^x. Logarithms to base 10 turn it into the line log N = log a + x log b, the line is fitted by least squares, and a and b are read back as powers of 10.