Step 18: Comparing Two Groups
- When you compare two groups, the thing you estimate is a difference: \( \mu_1 – \mu_2 \) for means, or \( p_1 – p_2 \) for proportions. The key question is always: is zero a plausible value for that difference?
- Check the design first. Independent samples are two separate groups of people; paired samples link each value in one group to a partner in the other (often the same person measured twice).
- Independent means use \( SE = \sqrt{s_1^2/n_1 + s_2^2/n_2} \) with df = smaller \( n – 1 \). Paired data is turned into one list of differences and analysed with one-sample methods. Two proportions use a pooled SE for the test and an unpooled SE for the interval.
- Exam point: an interval for the difference that contains 0 means “no significant difference”; one that excludes 0 means “significant”. Always report the interval, not just the p-value.
Why comparing two groups is about a difference
Imagine a company tests two versions of a web page. Version A is shown to 1,000 visitors and 110 of them buy something (11%). Version B is shown to a different 1,000 visitors and 140 buy (14%). The team is excited: B looks 27% better. Someone starts writing the announcement.
But hold on. Even if the two pages were exactly as good as each other, two random groups of visitors would not behave identically. Some gap would appear just by chance. So the real question is: how often would chance alone produce a gap as big as 3 percentage points?
Most practical questions in statistics are like this: two groups, and what you care about is the difference between them. The strategy is familiar. Estimate \( \mu_1 – \mu_2 \) with \( \bar{x}_1 – \bar{x}_2 \), find that estimate’s standard error, and then use the familiar interval and test machinery.
Here is the single idea that ties the whole step together. The null hypothesis is almost always “no difference”, which in symbols is \( \mu_1 – \mu_2 = 0 \). So everything reduces to one question: is zero a believable value for the difference? If a confidence interval for the difference contains zero, the groups are not significantly different. If it excludes zero, they are.
Independent or paired? Decide before you calculate
Two different procedures exist, and using the wrong one gives wrong answers. So the very first job is to identify the design.
- Independent samples are two separate, unrelated groups: 40 patients given a drug and a different 40 given a placebo. A value in one group tells you nothing about any particular value in the other. The groups need not be the same size.
- Paired samples link each observation in one group to a specific observation in the other. The most common case is measuring the same person twice (before and after a course). Another is deliberately matching people into pairs, for example twins or students of similar ability. The sample sizes must be equal.
One question settles it: does each value in group 1 have a natural partner in group 2? If yes, the design is paired. If no, it is independent.
Two means from independent samples
For two separate groups, the difference \( \bar{x}_1 – \bar{x}_2 \) estimates \( \mu_1 – \mu_2 \). Its standard error combines the variability of both groups. Notice something surprising: the means are subtracted, but the variances add. That is because each group brings its own uncertainty, and subtracting does not cancel uncertainty out.
The exact degrees of freedom for this test come from a complicated formula that software works out. By hand, the book uses a conservative rule: df = (smaller sample size) − 1. This gives a slightly wider interval, which errs on the side of caution.
Conditions: both samples are random and independent of each other, and each group is either roughly normal or has at least 30 observations.
Two teaching methods are compared. Method B: \( n = 36,\ \bar{x} = 78,\ s = 9 \). Method A: \( n = 36,\ \bar{x} = 72,\ s = 8 \). Build a 95% interval for \( \mu_B – \mu_A \) and test whether the methods differ at \( \alpha = 0.05 \).
We are 95% confident that method B’s true mean is between about 1.9 and 10.1 points higher than method A’s. The interval excludes zero and p < 0.05, so the difference is significant. The interval and the test agree, as they always must. But notice how wide the interval is: the true advantage could be a modest 2 points or a large 10.
Two means from paired samples
Paired data has a neat shortcut. For each pair, work out the difference, \( d = \text{after} – \text{before} \). Then throw away the original two columns. You are left with one list of differences, and you analyse it with the one-sample methods you already know.
Here \( \bar{d} \) and \( s_d \) are the mean and standard deviation of the differences, and \( n \) is the number of pairs, not the total number of measurements.
Sixteen students sit a test before and after a training course. The differences (after − before) have \( \bar{d} = 5.0 \) points and \( s_d = 4.0 \) points. Does the course improve scores? Use \( \alpha = 0.05 \) and give a 95% interval.
Reject \( H_0 \): there is very strong evidence the course raises scores, by somewhere between about 2.9 and 7.1 points on average.
Why does pairing work so well? The students’ scores might range from 40 to 95, but their improvements cluster tightly around 5. That small spread makes the standard error small and t large. When you can pair, pair.
Comparing two proportions
The logic for proportions is the same, with one twist: the interval and the test use different standard errors.
For the interval, each group uses its own sample proportion:
For the test, \( H_0 \) says the two proportions are equal. If they are equal, the best estimate of that shared value combines all the data. This is the pooled proportion:
Conditions: both samples random and independent, with at least 10 successes and 10 failures in each group.
Back to the web pages. Version B converts 140 of 1,000 visitors; version A converts 110 of 1,000. Test at \( \alpha = 0.05 \) whether the rates differ, and give a 95% interval.
Since 0.043 < 0.05, B converts at a significantly higher rate. But the interval tells the fuller story: the true gain could be anything from 0.1 to 5.9 percentage points. B is probably better, but the data cannot say whether the gain is tiny or worthwhile. The announcement was premature.
Reading the interval and test together, and choosing the procedure
Because the null value for a difference is zero, you can read the test result straight from a 95% interval:
| If the 95% interval for the difference… | Then at \( \alpha = 0.05 \)… | Report |
|---|---|---|
| contains 0 | fail to reject \( H_0 \) | No significant difference detected |
| lies entirely above 0 | reject \( H_0 \) | Group 1 significantly higher |
| lies entirely below 0 | reject \( H_0 \) | Group 1 significantly lower |
| excludes 0, but only just | reject \( H_0 \) | Significant, but the size is poorly pinned down |
By this stage of a course, the most common exam error is not arithmetic. It is choosing the wrong test. Two questions settle it: what kind of data is it, and how were the groups formed?
| Data | Design | Procedure | df |
|---|---|---|---|
| Quantitative | One group vs a claimed value | One-sample t | \( n – 1 \) |
| Quantitative | Two separate groups | Two-sample t | smaller \( n – 1 \) |
| Quantitative | Same or matched subjects twice | Paired t on the differences | pairs − 1 |
| Categorical | One group vs a claimed value | One-proportion z | none |
| Categorical | Two separate groups | Two-proportion z (pooled) | none |
Practise: 6 questions
Try each one on paper first, then check the worked answer.
EasyQ1. Independent or paired? (a) Blood pressure of 30 patients before and after a drug. (b) Salaries of 50 graduates from university X and 60 from university Y. (c) Left-hand and right-hand grip strength of 25 people. Show worked answer
(b) Independent: different people, and the groups are even different sizes.
(c) Paired: two measurements on the same person, so each left-hand value has a natural right-hand partner.
EasyQ2. Group 1: \( n = 25,\ \bar{x} = 52,\ s = 6 \). Group 2: \( n = 25,\ \bar{x} = 48,\ s = 8 \). Find the standard error of the difference and the t statistic. Show worked answer
Difference \( = 52 – 48 = 4 \)
\( t = 4 \div 2.0 = 2.00 \), with conservative df = 25 − 1 = 24.
MediumQ3. Using Q2 and \( t^{*} = 2.064 \), build the 95% interval for \( \mu_1 – \mu_2 \). Is the difference significant? Show worked answer
Interval \( = 4 \pm 4.13 = (-0.13,\ 8.13) \)
The interval contains zero, so the difference is not significant at the 5% level. Zero is a plausible value for the true difference.
MediumQ4. In a paired study of 25 people, the differences have \( \bar{d} = 3 \) and \( s_d = 5 \). Find the SE, t and df, then build a 95% interval using \( t^{*} = 2.064 \). Show worked answer
\( t = 3 \div 1.0 = 3.00 \), df = 25 − 1 = 24
Interval \( = 3 \pm 2.064(1.0) = (0.94,\ 5.06) \)
Zero is excluded, so the mean difference is significantly different from zero.
MediumQ5. A retailer tests a new checkout page. Old page: 220 purchases from 2,000 visitors. New page: 260 from 2,000. Compute the pooled proportion and z for a two-sided test. Show worked answer
Conditions: 220/1,780 and 260/1,740 successes/failures, all above 10.
\( \hat{p}_{pool} = 480 \div 4000 = 0.12 \)
\( SE = \sqrt{0.12 \times 0.88 \times (2/2000)} = \sqrt{0.0001056} = 0.01028 \)
\( z = 0.02 \div 0.01028 = 1.95 \)
The two-sided p-value is just above 0.05 (about 0.052), so at \( \alpha = 0.05 \) we fail to reject \( H_0 \). Advice: collect more data before rolling out or abandoning the new page.
ExamQ6. A clinic compares recovery times for two therapies. Therapy 1: \( n = 25,\ \bar{x} = 14 \) days, \( s = 5 \). Therapy 2: \( n = 25,\ \bar{x} = 17 \) days, \( s = 6 \). (a) Identify the design. (b) Compute the SE of the difference. (c) With df = 24 and \( t^{*} = 2.064 \), build a 95% interval for \( \mu_1 – \mu_2 \). (d) Do the therapies differ significantly? Justify. [6 marks] Show worked answer
(b) \( SE = \sqrt{25/25 + 36/25} = \sqrt{1 + 1.44} = \sqrt{2.44} = 1.562 \) [1 for method, 1 for answer].
(c) Difference \( = 14 – 17 = -3 \). Margin \( = 2.064 \times 1.562 = 3.22 \). Interval \( = -3 \pm 3.22 = (-6.22,\ 0.22) \) days [2].
(d) The interval contains zero, so the therapies do not differ significantly at the 5% level [1]. It only just includes zero, so a larger study would be worthwhile; this is not proof that the therapies are equally effective.
Check yourself
1. Twenty people have their reaction time measured before and after drinking coffee. Which procedure fits?
2. Why does the two-proportion test use a pooled proportion?
3. A 95% interval for \( \mu_1 – \mu_2 \) is \( (-2,\ 7) \). What do you conclude at \( \alpha = 0.05 \)?
Your score: 0 out of 3
The short version
- Comparing two groups means estimating a difference. The key question is whether zero is a plausible value for it.
- Identify the design first: independent (separate people) or paired (same or matched people). The wrong procedure loses power or is invalid.
- Independent means: \( SE = \sqrt{s_1^2/n_1 + s_2^2/n_2} \), df = smaller \( n – 1 \). Paired data: work out the differences and use one-sample methods with df = pairs − 1.
- Two proportions: pooled SE for the test, each group’s own \( \hat{p} \) for the interval; you need at least 10 successes and 10 failures per group.
- Always report the interval, not just the p-value, and never stop an A/B test early. The full chapter in Statistics Made Simple has 58 practice questions with a complete answer key.
Further reading on EconTutorials
Older in-depth articles on this topic from our blog: