Step 12: Choosing and Running Statistical Tests
MatchesQuantitative results chapter · see also the Statistics guide and paired vs independent t-tests
In 30 seconds
- A hypothesis test asks: if there were really no effect (\( H_0 \)), how surprising would my data be? The answer is the p-value.
- If \( p < 0.05 \), the result is called statistically significant: the data are unlikely under \( H_0 \).
- Choose the test from three facts: the type of outcome, the number of groups, and whether groups are independent or paired.
- Common choices: chi-square for two categorical variables, t-test for two group means, ANOVA for three or more, correlation for two numeric variables, and non-parametric versions when assumptions fail.
- Always report an effect size alongside \( p \): significance says an effect is probably real, effect size says whether it matters.
Video lesson coming soonA short animated lesson for this step will appear here.
The logic of a test
You suspect working students score lower. In your sample they do, by a few marks. But samples vary by chance: if you drew another sample, the gap might shrink or grow. A test asks how often a gap at least this big would appear by chance alone if, in the whole population, there were no difference at all.
Key termsThe p-value is the probability of getting a result at least as extreme as yours if \( H_0 \) were true. The significance level \( \alpha \) (usually 0.05) is the cut-off you set in advance. A Type I error is rejecting a true \( H_0 \); a Type II error is failing to reject a false one.
Which test? A decision table
| Question | Outcome | Groups / variables | Parametric test | Non-parametric alternative |
|---|---|---|---|---|
| Are two categorical variables related? | Categorical | Two categorical variables | Chi-square test of independence (Fisher’s exact test for small counts) | |
| Do two independent groups differ? | Numeric | 2 independent groups | Independent t-test | Mann–Whitney U |
| Did the same people change? | Numeric | 2 paired measurements | Paired t-test | Wilcoxon signed-rank |
| Do three or more groups differ? | Numeric | 3+ independent groups | One-way ANOVA | Kruskal–Wallis |
| Do two numeric variables move together? | Numeric | Two numeric variables | Pearson correlation | Spearman correlation |
| Does X predict Y, controlling for others? | Numeric (or binary) | Several predictors | Regression (Step 13) | |
Use the non-parametric version when the outcome is ordinal, or when groups are small and the data are strongly skewed or have outliers.
Chapter 12 of Research Methods Made Simple covers this in full, and Appendix A has full decision charts for choosing a test.Get the textbook · $15
Example 1: chi-square test
Worked example
Of 100 first-year students, 30 have a job; of 100 final-year students, 50 have a job. Is having a job related to year of study?
Step 1\( H_0 \): job status is independent of year. Overall, \( 80/200 = 40\% \) work.
Step 2Expected counts under \( H_0 \): 40 working and 60 not working in each year.
Step 3\( \chi^2 = \sum \dfrac{(O-E)^2}{E} = \dfrac{(30-40)^2}{40} + \dfrac{(70-60)^2}{60} + \dfrac{(50-40)^2}{40} + \dfrac{(50-60)^2}{60} = 2.5 + 1.67 + 2.5 + 1.67 = 8.33 \).
Step 4With 1 degree of freedom, \( p = 0.004 \). Reject \( H_0 \).
Step 5Report: “Final-year students were more likely to have a job than first-years (50% vs 30%), \( \chi^2(1, N = 200) = 8.33 \), \( p = 0.004 \).”
Example 2: independent t-test
Worked example
Ten students working 15+ hours have marks 62, 58, 71, 65, 60, 55, 68, 63, 59, 64. Ten non-working students have 70, 66, 74, 61, 72, 69, 75, 67, 71, 68. Do the means differ?
Step 1Means: workers 62.5 (SD 4.79), non-workers 69.3 (SD 4.11). Difference \( = 6.8 \) marks.
Step 2Welch’s t-test (which does not assume equal variances): \( t = -3.41 \), \( df = 17.6 \), \( p = 0.003 \).
Step 3Effect size, Cohen’s \( d = 6.8 / 4.46 = 1.52 \), using the pooled SD of 4.46. That is a large effect.
Step 4Report: “Students working 15 or more hours scored lower (M = 62.5, SD = 4.79) than non-working students (M = 69.3, SD = 4.11), \( t(17.6) = -3.41 \), \( p = 0.003 \), \( d = 1.52 \).”
Step 5Interpret with care: this is an association from a survey, not proof that work causes lower marks (Step 5).
These small illustrative samples keep the arithmetic visible; a real dissertation would use far more students.
Effect sizes and power
With a big enough sample, even a trivial difference becomes “significant”. Effect sizes tell you how big a difference is. Rough guides: Cohen’s \( d \) of 0.2 small, 0.5 medium, 0.8 large; correlation \( r \) of 0.1 small, 0.3 medium, 0.5 large. Power is the chance of detecting a real effect of a given size; studies are usually planned for 80% power. To detect a medium effect (\( d = 0.5 \)) between two groups at \( \alpha = 0.05 \) with 80% power you need about 64 per group.
In your dissertationFor every test report the test name, the statistic, degrees of freedom, the exact p-value, an effect size, and a plain-English sentence. Check the assumptions (normality, equal variances, expected counts of at least 5 for chi-square) and say what you did if they failed.
Common mistakeSaying “\( p = 0.03 \) means there is a 3% chance the null hypothesis is true.” The p-value assumes \( H_0 \) is true and gives the chance of data this extreme. Equally, \( p = 0.20 \) does not prove there is no effect; it means the evidence is not strong enough to reject \( H_0 \).
Practise until it sticks: Research Methods Made Simple Practice Questions has 1,004 questions, each with a worked answer.Get the practice book · $9
Practise: 6 questions
Try each one on paper first, then check the worked answer.
EasyQ1. Choose the test: (a) Is gender related to whether a student has a job? (b) Do marks differ between three faculties? Show worked answer
(a) Chi-square test of independence: two categorical variables. (b) One-way ANOVA (or Kruskal–Wallis if assumptions fail): numeric outcome, three independent groups.
EasyQ2. The same 40 students take a stress survey before and after exams. Which test compares the scores? Show worked answer
A paired t-test (or Wilcoxon signed-rank), because the same people are measured twice.
MediumQ3. Observed counts are 18 and 32 in one row; expected counts are 25 and 25. Calculate these two cells’ contribution to \( \chi^2 \). Show worked answer
\( (18-25)^2/25 + (32-25)^2/25 = 49/25 + 49/25 = 1.96 + 1.96 = 3.92 \).
MediumQ4. Two groups differ by 3 marks and the pooled SD is 12. Calculate Cohen’s \( d \) and interpret. Show worked answer
\( d = 3/12 = 0.25 \): a small effect, whatever the p-value.
MediumQ5. A study of 20,000 students finds \( p < 0.001 \) and \( r = 0.04 \) between commuting time and marks. What should you conclude? Show worked answer
The relationship is very unlikely to be zero, but it is tiny (\( r = 0.04 \)). With huge samples, trivial effects become significant. In practical terms commuting time explains almost none of the variation in marks.
ExamQ6. Explain the difference between statistical significance and practical importance, with an example. [5] Show worked answer
Statistical significance means the result is unlikely if \( H_0 \) were true [1]; it depends heavily on sample size [1]. Practical importance depends on the size of the effect, measured by an effect size such as \( d \) or \( r \) [1]. Example: with 20,000 students, a 0.5-mark difference may be significant but too small to matter [1]. So report both p-values and effect sizes, and judge importance in context [1].
Check yourself
1. To compare mean marks of two independent groups, use
Numeric outcome, two independent groups.
2. The non-parametric alternative to Pearson correlation is
Spearman works on ranks, so suits ordinal or skewed data.
3. A Type I error is
A false positive; its probability is \( \alpha \).
Your score: 0 out of 3
The textbook and the practice book together: the Research Methods Made Simple bundle for $19 (was $24).Both books · $19
The short version
- Choose the test from outcome type, number of groups, and independent vs paired.
- Chi-square, t-tests, ANOVA and correlation cover most dissertations; switch to non-parametric tests when assumptions fail.
- Report the statistic, df, exact p, an effect size and a plain sentence.
- Significant is not the same as important; association is not causation.
Studying more than one subject? The Complete Book Bundle includes every Economics, Statistics, Mathematics, Econometrics and Research Methods book.
Statistical Test Chooser · $5Decision flowchart, test lookup table and reporting templates. Free sample on the product page. Or get all 7 research kits for $19.
Want more questions like these, with worked answers?
The practice book has 1,004 more, from quick recall to exam style.