Statistics · Hypothesis Testing
The P-Value Explained
The most misunderstood concept in all of statistics — what p-values actually mean, what they don’t mean, and how to use them correctly.
The p-value is one of the most widely used — and most widely misunderstood — tools in statistics. Researchers, journalists, and students routinely misinterpret what p-values actually tell us. Getting this right is not just an academic exercise: misinterpreting p-values has contributed to a replication crisis across science, economics, and medicine.
📘 Key Term
The p-value is the probability of obtaining a test statistic at least as extreme as the one observed, assuming the null hypothesis (H₀) is true. It measures how surprising the data is if H₀ were correct — not the probability that H₀ is true.
The Decision Rule
Before conducting a test, researchers choose a significance level α (typically 0.05 or 5%):
p-value ≤ α
Reject H₀. The result is statistically significant. The data is unlikely to have occurred by chance alone under H₀.
p-value > α
Fail to reject H₀. Insufficient evidence against H₀. Note: this does NOT mean H₀ is true.
Worked Example: Wage Gap Test
A researcher tests whether a new training programme raises wages. H₀: no effect. The test returns p = 0.03.
Interpretation: If the programme had no effect, we would observe results this extreme only 3% of the time by random chance.
Since p = 0.03 < α = 0.05, we reject H₀ and conclude there is statistically significant evidence the programme raised wages.
💡 Key Insight
A small p-value tells you the data is unlikely under H₀. It does NOT tell you: (1) the probability H₀ is true, (2) the size or importance of the effect, or (3) that the result will replicate. Statistical significance is not the same as practical significance — a tiny wage increase can be statistically significant with a large enough sample.
What the P-Value Is NOT
❌ It is NOT the probability that H₀ is true
❌ It is NOT the probability the result occurred by chance
❌ It is NOT a measure of effect size or importance
❌ A p > 0.05 does NOT prove H₀ is true — it just fails to reject it
⚠️ Common Error
The most common mistake: saying a p-value of 0.03 means ‘there is a 97% probability the effect is real’ or ‘only a 3% probability the result happened by chance.’ Both are wrong. The p-value is calculated assuming H₀ is true — it cannot tell you the probability that H₀ is true or false.
Q1. A study tests whether a drug reduces recovery time. At α = 0.05, the test yields p = 0.08. State the conclusion and explain what the p-value means in context. [5 marks]
Answer: Since p = 0.08 > α = 0.05, we fail to reject H₀. There is insufficient statistical evidence at the 5% significance level to conclude the drug reduces recovery time. The p-value of 0.08 means: if the drug had no effect, we would observe results at least this extreme 8% of the time by random variation alone. This does not prove the drug has no effect — it simply means the evidence is not strong enough to reject that possibility at our chosen significance level.
References
1. Wasserstein, R.L. and Lazar, N.A. (2016) ‘The ASA’s Statement on p-Values’, The American Statistician, 70(2), pp. 129–133.
2. Moore, D.S., McCabe, G.P. and Craig, B.A. (2021) Introduction to the Practice of Statistics. W.H. Freeman.
3. Field, A. (2018) Discovering Statistics Using IBM SPSS Statistics. SAGE.
4. Nuzzo, R. (2014) ‘Statistical Errors’, Nature, 506, pp. 150–152.
5. Triola, M.F. (2022) Elementary Statistics. Pearson.