Step 21: Comparing Several Means with ANOVA

Why not just run lots of t-tests?

A school tries three teaching methods on three groups of students. Method A averages 70 in the exam, method B 78 and method C 86. The obvious plan is to compare them in pairs with the two-sample t-test: A against B, B against C, A against C.

That plan has a hidden flaw. Each test run at \( \alpha = 0.05 \) has a 5% chance of a false alarm. Run three of them and the chance that at least one shouts “significant”, even when all three methods are identical, is roughly 14%.

With \( k \) groups, the number of pairs is \( k(k-1)/2 \). Three groups need 3 tests, five need 10, ten need 45. If the tests were independent, the chance of at least one false alarm across \( c \) comparisons would be:

For 3 comparisons that is \( 1 – 0.95^3 \approx 0.14 \). For 10 comparisons it is about 0.40. With six groups (15 comparisons) it passes 50%. Run enough comparisons and something will always look significant.

ANOVA solves this by asking a single question, “are any of these means different?”, and answering it with one test at one significance level. The 5% risk stays at 5% however many groups there are.

The logic: between versus within

Why is a method for comparing means called analysis of variance? Because the way to judge whether means differ is to compare two kinds of variation.

• Between-group variation: how far the group means sit from the overall (grand) mean. This is the signal.
• Within-group variation: how much individuals scatter around their own group’s mean. This is background noise, present even if the groups are identical.

If the gaps between group means are large compared with the scatter inside groups, the differences look real. If the gaps are lost in the scatter, they look like chance.

Here is a striking example. Two studies both have three groups of 5 with means 68, 74 and 80. In the first, each group’s standard deviation is 2; in the second it is 9. The means are identical, yet the first study gives F = 45.0 (overwhelmingly significant) and the second gives F = 2.22 (not significant, p ≈ 0.15). The only difference is the spread inside the groups. That is why the means alone can never settle the question.

Sums of squares, mean squares and F

Each kind of variation is measured with a sum of squares, just as variance was built earlier in the book:

SSB adds up over a few groups, while SSW adds up over many individuals, so they cannot be compared directly. Dividing each by its degrees of freedom turns them into averages called mean squares. Their ratio is the F-statistic:

Here \( k \) is the number of groups and \( N \) is the total number of observations. F is a signal-to-noise ratio. If all groups really have the same mean, MSB and MSW estimate the same background variance, so F sits near 1. As the means pull apart, MSB grows and F rises. Only large values point to a difference, so, like chi-square, the test is always right-tailed. F has two df: \( (k-1,\ N-k) \).

The ANOVA table, and a complete example

Software shows everything in a standard layout. The within-group row is sometimes labelled “Error” or “Residual”.

Two checks catch most arithmetic slips: SSB + SSW must equal SST, and \( (k-1) + (N-k) \) must equal \( N-1 \).

ANOVA needs three conditions:

1. Independence: independent random samples, with observations independent within each group. This is a design matter that no plot can repair.
2. Normality: each group roughly normal. ANOVA copes with mild departures, especially when the groups are the same size.
3. Equal variances: a practical rule is that the largest group standard deviation should be no more than about twice the smallest.

In the teaching example every group’s standard deviation is about 3.16, so the ratio is 1.0, comfortably below 2.

What a significant F does not tell you, and how big the effect is

To find where the differences are, follow a significant ANOVA with a post hoc (“after the fact”) procedure. These make pairwise comparisons, but adjust the significance level so the family-wise error rate stays at 5%. The best known are Tukey’s HSD and the simpler, more cautious Bonferroni correction, which divides \( \alpha \) by the number of comparisons. Order matters: run the overall F first, and go on to post hoc tests only if it is significant.

For the teaching methods, Bonferroni tests each of the 3 pairs at \( 0.05 \div 3 \approx 0.0167 \). Each comparison uses \( SE = \sqrt{10(1/5 + 1/5)} = 2.0 \) with df = 12. A vs B and B vs C each differ by 8, so \( t = 4.00 \) (p ≈ 0.0018); A vs C differs by 16, so \( t = 8.00 \) (p < 0.001). All are below 0.0167, so now we can say all three methods differ, with C best and A worst.

Finally, significance is not the same as importance. With large samples, a tiny spread of means can give a significant F. The usual effect size is eta-squared, the share of the total variation explained by which group you are in:

For the teaching methods, \( \eta^2 = 640 \div 760 = 0.842 \): about 84% of the variation in scores is explained by the teaching method. A significant F with \( \eta^2 = 0.02 \) would describe a real but practically negligible difference.

One more link: with just two groups, ANOVA gives the same answer as the two-sample t-test, and \( F = t^2 \). With two groups, prefer the t-test, because it also gives a direction and an interval.

Practise: 6 questions

Try each one on paper first, then check the worked answer.

Check yourself

The short version

• Many pairwise t-tests inflate the family-wise error rate (about 14% with three groups). ANOVA asks one question with one test.
• ANOVA compares between-group signal with within-group noise: \( F = MSB/MSW \), with df \( (k-1,\ N-k) \). F near 1 supports \( H_0 \); large F rejects it.
• SS and df both add down the ANOVA table. Conditions: independence, roughly normal groups, largest SD no more than about twice the smallest.
• A significant F means at least one mean differs. Use Tukey or Bonferroni to find which, and report \( \eta^2 = SSB/SST \) for size.
• With two groups, ANOVA matches the t-test (\( F = t^2 \)). The full chapter in Statistics Made Simple has over 50 practice questions with a complete answer key.