Step 13: Hypothesis Testing: The Foundations
- A hypothesis test assumes the null hypothesis \( H_0 \) (“no effect”, always an equals sign) is true. It then asks whether the data would be too surprising if that were so. The alternative \( H_a \) is what you suspect instead.
- The test statistic = (estimate − hypothesised value) ÷ standard error: \( t \) for a mean (\( df = n – 1 \)), \( z \) for a proportion, with the standard error worked out from \( p_0 \).
- The p-value is the chance of a result at least as extreme as yours, assuming \( H_0 \) is true. If \( p \le \alpha \), reject \( H_0 \); otherwise “fail to reject”. Never say “accept”.
- Exam point: a Type I error rejects a true \( H_0 \) (probability \( \alpha \)); a Type II error misses a false one (probability \( \beta \)). Power \( = 1 – \beta \).
The logic of a test: assume, then look for surprise
A company says its batteries last 500 hours. A consumer group tests 16 of them and gets a mean of 486 hours, with a standard deviation of 20 hours. That is 14 hours short. Is the advert false? Maybe. Or maybe these 16 batteries were just a slightly unlucky sample, since Step 9 showed that sample means always bounce around. We need a fair way to tell these two stories apart.
The method feels backwards at first. You assume the claim you doubt is true, then check whether the data makes that assumption look silly. It works like a criminal trial. The court presumes the defendant is innocent and asks whether the evidence is too strong to fit with innocence. If it is, the court convicts. If not, the defendant is acquitted. But acquittal does not prove innocence; it only means guilt was not shown. Tests behave exactly the same way.
| Situation | \( H_0 \) | \( H_a \) | Type |
|---|---|---|---|
| Batteries last less than advertised | \( \mu = 500 \) | \( \mu < 500 \) | Left-tailed |
| A new method raises scores | \( \mu = 70 \) | \( \mu > 70 \) | Right-tailed |
| The mean differs from target either way | \( \mu = 100 \) | \( \mu \ne 100 \) | Two-tailed |
| Support differs from half | \( p = 0.5 \) | \( p \ne 0.5 \) | Two-tailed |
The test statistic
Once the hypotheses are set, you measure how far the data is from what \( H_0 \) predicts, in standard-error units. That distance is the test statistic.
For a mean when \( \sigma \) is unknown, this gives the one-sample \( t \) statistic with \( df = n – 1 \). For a proportion you use \( z \). Note that its standard error uses the hypothesised \( p_0 \), not \( \hat{p} \), because the whole test runs on the assumption that \( H_0 \) is true. (This is different from the confidence interval in Step 10, which uses \( \hat{p} \).)
The p-value and the decision
The test statistic says how far out the data landed. The p-value turns that distance into a probability.
Think of it as a measure of surprise. A small p-value means data like yours would be rare if \( H_0 \) were true, which counts as evidence against \( H_0 \). For a two-tailed test, “extreme” means far out in either direction, so the p-value is usually twice the one-tail area.
To make a decision, you pick a significance level \( \alpha \) in advance, most often 0.05, or 0.01 when mistakes are costly.
- If \( p \le \alpha \): reject \( H_0 \). The result is “statistically significant”.
- If \( p > \alpha \): fail to reject \( H_0 \).
The words “fail to reject” are chosen with care. Like the acquitted defendant, a null hypothesis that survives has not been proved true. The evidence was just not strong enough. A small study can easily miss an effect that really exists.
Test the battery claim at \( \alpha = 0.05 \): \( n = 16 \), \( \bar{x} = 486 \), \( s = 20 \). Battery life is assumed roughly normal, which matters because \( n < 30 \).
If the batteries really averaged 500 hours, a sample of 16 this low would happen only about 0.7% of the time. There is convincing evidence that the true mean battery life is less than 500 hours. A two-tailed test would double the p-value to about 0.014.
Do not treat 0.05 as a cliff edge. A machine part should average 100 mm; a sample of 36 gives \( \bar{x} = 104 \), \( s = 12 \), so \( t = 4/2 = 2.00 \) and the two-tailed \( p \approx 0.053 \). You fail to reject at 5%. But 0.053 and 0.048 describe almost the same evidence. Report the actual p-value and let readers judge.
Testing a proportion
The same four steps work for proportions. The conditions are a random sample with \( np_0 \ge 10 \) and \( n(1-p_0) \ge 10 \).
A company claims 65% of its customers are satisfied. A survey of 100 random customers finds 75 satisfied. Is there evidence that satisfaction is above 65%? Use \( \alpha = 0.05 \).
There is convincing evidence that more than 65% of customers are satisfied.
Errors, power and the link with intervals
Because you decide from a sample, you can be wrong in two different ways.
| \( H_0 \) is actually true | \( H_0 \) is actually false | |
|---|---|---|
| Reject \( H_0 \) | Type I error (probability \( \alpha \)) | Correct decision (power) |
| Fail to reject \( H_0 \) | Correct decision | Type II error (probability \( \beta \)) |
A Type I error is a false alarm, such as deciding a drug works when it does not. In the trial, it convicts an innocent person. A Type II error is a missed detection, such as deciding a drug that really works shows nothing. It acquits a guilty person. Lowering \( \alpha \) from 0.05 to 0.01 makes false alarms rarer, but it also makes real effects easier to miss, so \( \beta \) rises. The right \( \alpha \) depends on which mistake would be worse. For screening a serious disease, missing it is worse, so a higher \( \alpha \) can be sensible.
Power rises with a larger sample (the main lever you control), a bigger true effect, less variability in the data, and a larger \( \alpha \). A study with low power will probably miss the very effect it was meant to find, so a non-significant result from it tells you very little.
Intervals and tests agree. A value lies outside a 95% confidence interval exactly when a two-sided test of that value is rejected at \( \alpha = 0.05 \). The battery interval from Step 10 was (475.3, 496.7), which excludes 500, matching the test. The test asks “is this one value believable?” The interval asks “which values are believable?” and shows how big the effect might be, so report both.
Finally, “statistically significant” only means the effect is unlikely to be zero. It does not mean the effect matters. With 100,000 people, a drug that cuts weight by 0.1 kg on average could give \( p < 0.001 \): real, but useless in practice. Always ask two questions: is it real, and is it big enough to care about?
Practise: 6 questions
Try each one on paper first, then check the worked answer. Use the z-table from Step 8 and the t-table from Step 10.
EasyQ1. A delivery firm claims its mean delivery time is 30 minutes; you suspect it is longer. A bottling machine should fill 500 ml, and any change either way is a problem. Write \( H_0 \) and \( H_a \) for each, and say which is one-tailed. Show worked answer
Bottling: \( H_0: \mu = 500 \), \( H_a: \mu \ne 500 \). This is two-tailed.
EasyQ2. A test gives \( p = 0.03 \). State the decision at \( \alpha = 0.05 \) and at \( \alpha = 0.01 \). Show worked answer
At \( \alpha = 0.01 \): \( 0.03 > 0.01 \), so fail to reject \( H_0 \).
The same evidence can lead to different decisions depending on the level you chose in advance.
MediumQ3. \( \bar{x} = 22 \), \( \mu_0 = 25 \), \( s = 6 \), \( n = 25 \). Find \( t \) and its degrees of freedom. Show worked answer
\( t = (22 – 25)/1.2 = -3/1.2 = -2.50 \).
\( df = 25 – 1 = 24 \).
MediumQ4. A poll of 400 voters finds 55% in favour. Test \( H_0: p = 0.50 \) against \( H_a: p \ne 0.50 \) at \( \alpha = 0.05 \). Show worked answer
\( SE = \sqrt{0.5 \times 0.5/400} = \sqrt{0.000625} = 0.025 \).
\( z = (0.55 – 0.50)/0.025 = 2.00 \).
Two-tailed p-value \( = 2 \times (1 – 0.9772) = 0.0456 \).
\( 0.0456 < 0.05 \), so reject \( H_0 \): there is evidence that support differs from 50%.
MediumQ5. A test has power 0.60. (a) What is \( \beta \)? (b) What happens to Type I and Type II error rates if \( \alpha \) is lowered from 0.05 to 0.01? Show worked answer
(b) The Type I error rate falls to 1%. Because stronger evidence is now needed, real effects are missed more often, so \( \beta \) rises and power falls.
ExamQ6. A cereal maker claims boxes average 350 g. A consumer group weighs 25 random boxes and finds \( \bar{x} = 344 \) g and \( s = 10 \) g; the data is roughly symmetric. (a) State hypotheses to test whether boxes are underfilled. [2] (b) Check the conditions. [1] (c) Calculate the test statistic and df. [2] (d) The one-tailed 5% critical value at df = 24 is 1.711. State and interpret the decision. [2] (e) Which type of error could you have made? [1] Show worked answer
(b) Random sample, and the data is roughly symmetric, which is acceptable for \( n = 25 \) [1].
(c) \( SE = 10/\sqrt{25} = 2 \) [1]; \( t = (344 – 350)/2 = -3.00 \), \( df = 24 \) [1].
(d) \( |-3.00| > 1.711 \), so reject \( H_0 \) [1]. There is convincing evidence that the mean box weight is below the claimed 350 g [1].
(e) Because we rejected \( H_0 \), the only possible mistake is a Type I error (rejecting a true claim) [1].
Check yourself
1. What does a p-value of 0.02 mean?
2. A test concludes a drug works when it actually does not. Which error is this?
3. A 95% confidence interval for \( \mu \) is (12, 18). Would a two-sided test of \( \mu = 15 \) be rejected at \( \alpha = 0.05 \)?
Your score: 0 out of 3
The short version
- Assume \( H_0 \) (no effect, always with an equals sign) and ask whether the data is too surprising to fit with it. Choose \( H_a \) from the question, before you see the data.
- Test statistic = (estimate − hypothesised value) ÷ standard error: \( t \) for a mean with \( df = n – 1 \); \( z \) for a proportion with the SE built from \( p_0 \).
- The p-value is the chance of a result at least this extreme if \( H_0 \) is true. If \( p \le \alpha \), reject \( H_0 \); otherwise fail to reject. Never “accept”, and always conclude in context.
- Type I error (probability \( \alpha \)) rejects a true \( H_0 \); Type II error (probability \( \beta \)) misses a false one. Power \( = 1 – \beta \) rises with sample size. A value outside a 95% interval is rejected by a two-sided 5% test, and statistical significance is not the same as practical importance.
- The full chapter in Statistics Made Simple has many more worked examples and a full set of tiered practice questions with a complete answer key.
Further reading on EconTutorials
Older in-depth articles on this topic from our blog: