Step 14: Errors and Power in Significance Tests
- A significance test can go wrong in two ways. A Type I error rejects a null hypothesis that is actually true. A Type II error fails to reject a null hypothesis that is actually false.
- The probability of a Type I error (when \(H_0\) is true) is the significance level \(\alpha\). The probability of a Type II error is called \(\beta\).
- Power \(= 1 – \beta\) is the probability that the test correctly rejects a false \(H_0\). Power goes up with a larger sample, a larger \(\alpha\), a bigger true effect and less variability.
- Exam marks come from describing each error in context and naming a realistic consequence, not just reciting “reject a true \(H_0\)”.
Two ways a test can be wrong
Think about the smoke alarm in your kitchen. Its job is to decide between two possibilities: “there is no fire” and “there is a fire”. It can go off when you are only making toast (a false alarm), or stay silent while something is really burning (a missed fire). Both are mistakes, with very different consequences.
A significance test is like that alarm. You start by assuming the null hypothesis \(H_0\) is true (“no fire”, “no change”, “the claim is correct”). If the data would be very surprising under \(H_0\), you reject it in favour of the alternative \(H_a\). Because you only see a random sample, chance will sometimes fool you.
There are four possible situations. Two are correct decisions and two are errors:
| \(H_0\) is actually true | \(H_0\) is actually false | |
|---|---|---|
| Reject \(H_0\) | Type I error (probability \(\alpha\)) | Correct decision (probability = power) |
| Fail to reject \(H_0\) | Correct decision | Type II error (probability \(\beta\)) |
A Type I error is a false positive; a Type II error is a false negative. A court trial works the same way. \(H_0\) is “the defendant is innocent”. Convicting an innocent person is a Type I error. Letting a guilty person go free is a Type II error.
Describing errors in context, with consequences
Examiners give you a situation and ask you to describe the error in that situation, often with a consequence. The safe method is to take the general definition and replace “\(H_0\)” with what the null hypothesis actually says.
A phone company says that 80% of its customers are satisfied. A consumer group suspects the true figure is lower. It surveys a random sample of customers and tests \(H_0: p = 0.80\) against \(H_a: p < 0.80\), where \(p\) is the proportion of all the company’s customers who are satisfied. Describe a Type I and a Type II error in context, with a consequence of each.
Each description says what was concluded and what was really true, in context.
Two more common contexts:
- Medical screening. \(H_0\): the patient does not have the disease. A Type I error tells a healthy person they may be ill, causing worry and extra tests. A Type II error tells an ill person they are fine, so treatment is delayed. The Type II error is usually far more serious here.
- Quality control. \(H_0\): the machine is working properly. A Type I error stops a good machine for repairs, wasting time and money. A Type II error leaves a faulty machine running, so defective products reach customers.
Power: the chance of catching a real effect
If \(H_0\) really is false, you want your test to notice. The probability that it does is called the power of the test. Since a false \(H_0\) is either detected (rejected) or missed (Type II error), these two probabilities add up to 1.
Power always depends on which alternative is true. A test might have only a 30% chance of detecting a tiny difference but a 95% chance of detecting a large one. A low-power study will probably miss the effect it was looking for.
For example, a coin is suspected of landing heads too often. You toss it \(n = 100\) times and test \(H_0: p = 0.5\) against \(H_a: p > 0.5\) at \(\alpha = 0.05\). Suppose the true probability of heads is actually 0.6. Find the power of the test.
If the coin really lands heads 60% of the time, a test with 100 tosses has about a 64% chance of detecting it and about a 36% chance of missing it. With \(n = 200\) tosses the same calculation gives power of about 0.89.
AP Statistics does not usually ask you to calculate power, but it does expect you to explain what changes it.
What increases power?
There are four levers. Anything that pulls the null and alternative distributions apart, or makes them narrower, raises power.
| Change | Effect on power | Why |
|---|---|---|
| Larger sample size \(n\) | Increases | Smaller standard deviation of the statistic, so the curves overlap less. This is the lever researchers control best. |
| Larger significance level \(\alpha\) | Increases | The cut-off moves closer to the null value, so rejecting is easier. The price is more Type I errors. |
| Larger effect size (true value further from the null value) | Increases | A big difference is easier to spot than a small one. |
| Less variability in the data | Increases | Less noise, narrower curves. Better measurement or a more uniform population helps. |
In the coin example, raising \(\alpha\) from 0.05 to 0.10 lifts the power from about 0.64 to about 0.77.
The trade-off between the two errors
For a fixed sample size, you cannot make both errors rarer at the same time. Lowering \(\alpha\) from 0.05 to 0.01 makes false alarms rarer, but it demands stronger evidence before you reject. That means real effects are missed more often, so \(\beta\) goes up and power goes down. Raising \(\alpha\) does the opposite. To reduce both, collect more data.
So how do you choose \(\alpha\)? Ask which error has worse consequences:
- If a Type I error is more serious (for example, approving a new drug that does not work and has side effects), use a smaller \(\alpha\) such as 0.01.
- If a Type II error is more serious (for example, missing a dangerous fault in aircraft parts, or missing a disease in screening), use a larger \(\alpha\) such as 0.10 to raise power.
The value 0.05 is a convention, not a law of nature.
Practise: 6 questions
Try each one on paper first, then check the worked answer.
EasyQ1. A smoke alarm treats \(H_0\) as “there is no fire”. Describe a Type I error and a Type II error for the alarm. Show worked answer
Type II error: the alarm stays silent (fails to reject “no fire”) when there really is a fire. This is a missed fire, and it is clearly the more dangerous error, which is why alarms are designed to be sensitive.
EasyQ2. A test uses \(\alpha = 0.05\). For a particular alternative, \(\beta = 0.20\). What is the probability of a Type I error if \(H_0\) is true? What is the power? Show worked answer
Power \(= 1 – \beta = 1 – 0.20 = 0.80\).
So if this alternative is true, the test has an 80% chance of correctly rejecting \(H_0\).
MediumQ3. A researcher rejects \(H_0\) at the 5% level. Which type of error might she have made? Could she have made the other type? Show worked answer
She cannot have made a Type II error, because a Type II error requires failing to reject \(H_0\).
MediumQ4. State whether each change increases, decreases or does not change power: (a) \(n\) from 50 to 200; (b) \(\alpha\) from 0.05 to 0.01; (c) the true value moves further from the null value. Show worked answer
(b) Decreases. A smaller \(\alpha\) needs stronger evidence to reject, so \(\beta\) rises and power falls.
(c) Increases. A larger effect is easier to detect.
MediumQ5. A blood test screens for a serious but treatable disease, with \(H_0\): the patient does not have the disease. Which error is more serious? Should the designers use \(\alpha = 0.01\) or \(\alpha = 0.10\)? Show worked answer
To reduce the chance of a Type II error, use the larger \(\alpha = 0.10\). This raises power (lowers \(\beta\)), at the cost of more false alarms.
ExamQ6. A factory’s machine should produce at most 5% defective bolts. Each day an inspector takes a random sample and tests \(H_0: p = 0.05\) against \(H_a: p > 0.05\), where \(p\) is the proportion of all bolts made that day that are defective. If \(H_0\) is rejected, the machine is stopped for repair. (a) Describe a Type I error in context and give a consequence. [2 marks] (b) Describe a Type II error in context and give a consequence. [2 marks] (c) The power of the test against \(p = 0.10\) is 0.62. Interpret this value and give one way to increase it without changing \(\alpha\). [2 marks] Show worked answer
(b) Not finding convincing evidence that more than 5% are defective when in fact more than 5% are [1]. Faulty bolts are sold to customers, causing complaints or safety problems [1].
(c) If 10% of bolts are really defective, there is a 0.62 probability that the test will correctly reject \(H_0\) and stop the machine [1]. Increase the sample size, which reduces the standard deviation of \(\hat{p}\) and so raises power [1].
Check yourself
1. A Type II error is:
2. If \(\beta = 0.15\), the power of the test is:
3. Which change raises power without raising the chance of a Type I error?
Your score: 0 out of 3
The short version
- Type I error: reject a true \(H_0\) (false alarm), probability \(\alpha\). Type II error: fail to reject a false \(H_0\) (missed effect), probability \(\beta\).
- Always describe errors in context: the decision, the truth, and the parameter in words, plus a real consequence.
- Power \(= 1 – \beta\) is the chance of detecting a real effect. It depends on which alternative is true.
- Power rises with larger \(n\), larger \(\alpha\), larger effect size and less variability. Larger \(n\) is the best lever.
- For fixed \(n\), lowering \(\alpha\) raises \(\beta\). Choose \(\alpha\) by asking which error is worse.
Further reading on EconTutorials
Older in-depth articles on this topic from our blog: