Type One and Type Two Errors

Two ways to be wrong

A test ends in one of two verdicts, reject $H_0$H₀ or do not reject it, and $H_0$H₀ is in fact either true or false. That gives four outcomes, and two of them are errors.

Rejecting $H_0$H₀ when it is true is a type one error: a false alarm. Not rejecting $H_0$H₀ when it is false is a type two error: a real effect missed. The other two outcomes, rejecting a false $H_0$H₀ and keeping a true one, are correct decisions.

Take a machine that fills cereal boxes with a mean of 500 g and a standard deviation of 12 g. A sample of 144 boxes is weighed to test $H_0$H₀: $\mu =500$μ = 500 against $H_1$H₁: $\mu >500$μ > 500, at the 5 percent level. A type one error stops a machine that was filling correctly. A type two error leaves a machine running that has started to overfill.

The chance of a type one error

The standard error is $12÷\sqrt{144}=12/12=1\text{g}$12 ÷ √144 = 12 / 12 = 1 g, so $z=\left(\bar{x}-500\right)/1=\bar{x}-500$z = (x̄ − 500) / 1 = x̄ − 500. The critical region is $z\ge 1.645$z ≥ 1.645, which is $\bar{x}\ge 501.645\text{g}$x̄ ≥ 501.645 g.

A type one error can only happen when $H_0$H₀ is true. Then z follows the standard normal distribution, and the chance it lands in the critical region is $P\left(Z\ge 1.645\right)=0.05$P(Z ≥ 1.645) = 0.05. So the probability of a type one error is the significance level itself, chosen before the test. It is written $\alpha$α.

The gold curve is the distribution of z if $H_0$H₀ is true. The shaded area beyond 1.645 is the chance of rejecting $H_0$H₀ when it is true, the type one error, $\alpha =0.05$α = 0.05.

The chance of a type two error

A type two error can only happen when $H_0$H₀ is false, so its probability depends on what the true mean is. Suppose the machine has drifted to a mean of 503 g. Then x̄ is normal with mean 503 and standard error 1, so z = x̄ − 500 is normal with mean 3 and standard deviation 1.

The test fails to reject when z < 1.645, the same line as before. The probability of that, written $\beta$β, is the area under the true distribution to the left of the line: $\beta =P\left(z<1.645\right)=\Phi \left(1.645-3\right)=\Phi \left(-1.355\right)=0.0877$β = P(z < 1.645) = Φ(1.645 − 3) = Φ(−1.355) = 0.0877.

So about 9 times in 100, a sample from this overfilling machine would pass the test. The chance of detecting the drift, $1-\beta =0.9123$1 − β = 0.9123, is called the power of the test.

The gold curve is the distribution of z under $H_0$H₀, centered at 0. The dashed curve is its distribution when the true mean is 503 g, centered at 3. The shaded area under the dashed curve, to the left of 1.645, is $\beta =0.0877$β = 0.0877: results from the true distribution that fall short of the line.

Demanding more evidence

To make false alarms rarer, lower the significance level. At 1 percent the critical value moves out to 2.326, so $\alpha =0.01$α = 0.01. The line has moved toward the true distribution, and more of it now falls short: $\beta =\Phi \left(2.326-3\right)=\Phi \left(-0.674\right)=0.2502$β = Φ(2.326 − 3) = Φ(−0.674) = 0.2502.

Move the line further, to 2.5, and $\alpha =P\left(Z\ge 2.5\right)=0.0062$α = P(Z ≥ 2.5) = 0.0062 while $\beta =\Phi \left(2.5-3\right)=\Phi \left(-0.5\right)=0.3085$β = Φ(2.5 − 3) = Φ(−0.5) = 0.3085. With the sample size fixed, every move of the line that shrinks one error grows the other.

The same two curves with the line moved out to 2.5. The tail of the gold curve beyond the line is now only 0.0062, but the shaded area under the dashed curve has grown to $\beta =0.3085$β = 0.3085.

A larger sample lowers both

The sample size changes how far apart the two curves sit. With 36 boxes instead of 144, the standard error is $12÷\sqrt{36}=2\text{g}$12 ÷ √36 = 2 g, so a true mean of 503 g puts the center of z at (503 − 500) ÷ 2 = 1.5 instead of 3.

At the 5 percent level $\alpha$α is still 0.05, but $\beta =\Phi \left(1.645-1.5\right)=\Phi \left(0.145\right)=0.5576$β = Φ(1.645 − 1.5) = Φ(0.145) = 0.5576: more than half the time, the test would miss the drift. With 144 boxes it was 0.0877. A larger sample separates the two curves, which is the one way to reduce $\beta$β without raising $\alpha$α.

The usual mistakes

Swapping the two names. Type one is rejecting a true $H_0$H₀, the false alarm. Type two is keeping a false $H_0$H₀, the miss.

Taking $\beta$β as $1-\alpha$1 − α. 1 − 0.05 = 0.95 is the chance of keeping $H_0$H₀ when $H_0$H₀ is true, a correct decision. $\beta$β is worked out under the true mean, with $H_0$H₀ false.

Shading under the wrong curve. $\beta$β is an area under the true distribution, the one centered at 3, not under the $H_0$H₀ curve.

Reading $\alpha$α as the chance that $H_0$H₀ is true after a rejection. $\alpha$α is the chance of rejecting, given that $H_0$H₀ is true.

Loaves that are light

In the application below, an inspector weighs 25 loaves to test whether their mean is below 400 g. The critical region is found under $H_0$H₀, and the chance of a type two error is then found under the true mean of 395 g, as for the cereal.