Sampling and Inference

Inference uses a sample to make a claim about a whole population. Every method below answers one question: how far can a sample be trusted to speak for the population it came from?

What is a sample, and what makes a good one?

A population is the whole group you want to know about, and a sample is the part of it you measure. A number that describes the population is a parameter. The same number computed from the sample is a statistic, and it is your estimate of the parameter. See Populations and Samples.

A good sample gives every member a known chance of being chosen. In a simple random sample every member has the same probability. In a systematic sample you take every tenth name on a list, which relies on the order of the list being fair. See Sampling Methods.

In an experiment the researcher assigns the treatment, for example by a coin toss. In an observational study people put themselves into groups, and those groups can differ before the treatment starts. Only random assignment supports a claim about cause. See Observational Studies and Experiments.

Capture and recapture estimates a population that cannot be listed. Tag 60 fish and release them. Later catch 80 fish and find 12 tagged. If the tagged fish have mixed fully, the same fraction of the lake is tagged: $\frac{60}{N}=\frac{12}{80}$60/N = 12/80, so $N=60\times 80/12=400$N = 60 × 80 / 12 = 400. See Capture and Recapture.

Why is the mean of a sample so well behaved?

Every sample of the same size gives a different mean, so the sample mean has a distribution of its own, called the sampling distribution. See The Sampling Distribution.

Its mean is the population mean $\mu$μ. Its variance is $\frac{{\sigma }^2}{n}$σ²/n, because n independent variances add to $n{\sigma }^2$nσ² and dividing the total by n divides the variance by $n^2$n². The square root, $\frac{\sigma }{\sqrt{n}}$σ/√n, is the standard error: the standard deviation of the sample mean. See Mean and Variance of X-bar and Standard Error.

With $\sigma =20$σ = 20 and n = 100, the standard error is $20÷\sqrt{100}=2$20 ÷ √100 = 2. Individual values scatter by about 20, and their average by about 2. To halve the standard error you need four times as many readings.

The central limit theorem supplies the shape. Whatever the population looks like, the sample mean is approximately normal once n is large enough. About thirty is usually enough, and a badly lopsided population needs more. See The Central Limit Theorem and Applying the Theorem.

The same approximation works for the binomial when np and n(1 − p) are both greater than 5: a normal curve with mean np and variance np(1 − p) sits on top of the bars. The bars have width, so a count of 40 covers 39.5 to 40.5. See Approximating a Binomial and Continuity Correction.

How do you estimate a population value from a sample?

The sample mean estimates the population mean, and a single number like this is a point estimate. Unbiased does not mean right; it means right on average over many samples. See Point Estimates.

Variance needs a correction. The data sits closer to its own mean than to the population mean, so dividing by n understates the spread on average. Divide by n − 1 instead. See Unbiased Sample Variance.

A confidence interval reports a range instead. Take the estimate, then add and subtract 1.96 standard errors, because z = 1.96 leaves 2.5 percent in each tail of the normal distribution.

The 95 percent describes the method: repeat it many times and 95 in 100 of the intervals contain the true mean. The one in front of you either contains it or does not. See Confidence Intervals.

How does a hypothesis test work?

Assume nothing has changed, work out how surprising your data would be if that were true, and reject the assumption if the data is surprising enough.

The null hypothesis, $H_0$H₀, claims no effect. The alternative, $H_1$H₁, claims an effect and needs evidence before you believe it. A test can reject the null hypothesis or fail to reject it; it can never prove the null true. See Null and Alternative Hypotheses.

Decide before seeing the data which tail counts. A one-tailed test puts the whole 5 percent in one tail. A two-tailed test splits it, 2.5 percent in each tail, so its critical value sits further out. See One-Tailed and Two-Tailed.

The z-test handles a mean when $\sigma$σ is known: $z=\left(\bar{x}-\mu \right)/\left(\sigma ÷\sqrt{n}\right)$z = (x̄ − μ) / (σ ÷ √n). Subtract the claimed mean, divide by the standard error, and compare z with the critical value. See The Z-Test.

A critical region fixes the boundary in advance: mark off the most extreme 5 percent, and any result landing there rejects the null. A p-value shades everything at least as extreme as the result you saw and reports that area as a probability. See Critical Regions and P-Values.

For a proportion from a small sample, use the binomial directly: if a die is fair, sixes in 60 rolls follow $B\left(60,\frac{1}{6}\right)$B(60, 1/6), and the probability of 17 or more sixes is the p-value. See Testing a Proportion with the Binomial.

A type one error rejects a null hypothesis that was true, and its probability is the significance level you chose. A type two error fails to reject a false one. Demand more evidence and the first rate falls while the second rises. See Type One and Type Two Errors.

Which test fits which question?

The chi-squared tests compare observed counts with expected counts. In a two-way table, the expected count for a cell is the row total times the column total divided by the grand total. Square each gap between observed and expected, divide by the expected count, and add up every cell. The degrees of freedom are (rows − 1) × (columns − 1). See Expected Frequencies and The Chi-Squared Test for Independence.

The goodness of fit test uses the same statistic against a claimed distribution: a fair die rolled 60 times expects ten of each face. See The Chi-Squared Goodness of Fit Test.

When $\sigma$σ must be estimated from the sample, use t instead of z. The one-sample t-test divides the gap between the sample mean and the claimed mean by $s÷\sqrt{n}$s ÷ √n, with n − 1 degrees of freedom. The two-sample t-test divides the gap between two sample means by the standard error of that gap. See The One-Sample T-Test and The Two-Sample T-Test.

Spread can be tested too. Scale a sample variance by $\left(n-1\right)/{\sigma }^2$(n − 1) / σ² and it follows a chi-squared curve with n − 1 degrees of freedom. To compare two variances, divide the larger by the smaller; that ratio is F. See Testing a Variance with Chi-Squared and Comparing Two Variances with the F-Test.

The mistakes worth naming

• Reading a p-value backwards. It is the probability of the data given the null hypothesis, never the probability of the null hypothesis.
• Choosing the tail after seeing the data. That doubles the true false-alarm rate.
• Treating "not significant" as "no effect". A small sample often gives too little evidence to reject anything.
• Dividing by n for a sample variance. Divide by n − 1.
• Dropping the continuity correction. A count of 40 covers 39.5 to 40.5.
• Believing a big sample fixes a biased one. Size reduces random error and does nothing to systematic error.

Where this leads next

The binomial and normal distributions are built in the sets, counting and distributions guide. The mean and variance being tested come from the averages and spread guide. The test for a correlation coefficient is in the correlation and regression guide.

Learn this properly in the app

Math Challenge teaches each of these as an illustrated lesson with practice questions on the same skill.