- Sampling bias is a systematic error: the way a sample is chosen or collected makes some groups more likely to be counted than others, so the estimate misses the truth in a predictable direction.
- The three types you must name are undercoverage (some people cannot be selected), nonresponse (selected people do not answer) and voluntary response (people choose themselves).
- In our gym survey, 30% of students really use the gym, but the survey reports 40%, because gym users answered at 70% and non-users at only 45%.
- A bigger sample does not cure bias. It only makes the wrong answer more precise, as the 1936 Literary Digest poll showed with about 2.4 million replies.
- Exam point: name the bias, say who is missing or over-represented, and state the direction of the error in context.
The case: a gym survey that overstates gym use
A university wants to know how many of its 2,000 undergraduates use the campus gym at least once a week. The answer will decide whether the gym gets a bigger budget next year. The sports office emails a short survey to every student. It asks one question: “Did you use the gym at least once a week this term?”
Suppose we know the truth from the gym’s card-swipe records: 600 students, or 30%, are weekly users. The other 1,400 are not. Now look at who answers the email. Gym users care about the gym, so 70% of them reply. Non-users have little reason to open an email from the sports office, so only 45% of them reply.
That gives 420 replies from users (70% of 600) and 630 replies from non-users (45% of 1,400). In total 1,050 students reply. Among those replies, 420 say “yes”. The survey estimate is therefore \( 420 / 1050 = 0.40 \), or 40%.
The survey says 40%. The truth is 30%. The error is 10 percentage points, and it is not bad luck. Every student was contacted, the sample is large, and the arithmetic is right. The problem is who chose to answer. This post explains how to spot that kind of error, give it the right name and work out how big it is. (The figures are illustrative.)
The concept: sampling bias and its three main forms
A sample is useful only if it looks like the population on the thing we are measuring. When the method of choosing or collecting the sample makes some members of the population more likely to end up in the data than others, the sample is biased. The word “bias” in statistics does not mean prejudice. It means a systematic error: if you repeated the same method again and again, the estimates would centre on the wrong value.
That is different from sampling variability, the random scatter between one sample and the next. Random scatter shrinks as the sample gets bigger. Bias does not. Collect a million replies with a biased method and you get a very precise estimate of the wrong number.
Most sampling bias you will meet at school level falls into three groups.
Undercoverage happens when some members of the population have no chance, or a much smaller chance, of being selected. The usual cause is a poor sampling frame, the list we sample from. A survey that samples from a list of landline numbers misses people who only have a mobile. A survey of “shoppers” taken in a mall at 11 a.m. on a weekday misses people at work.
Nonresponse happens when people who were properly selected cannot be reached or refuse to answer, and those who do not answer differ from those who do. Our gym survey is a nonresponse problem: the frame (all 2,000 students) was perfect, but the replies were not.
Voluntary response happens when people select themselves, for example by calling a phone-in, clicking an online poll or leaving a review. The people with the strongest feelings, usually negative ones, are the most likely to take part, so the sample is not chosen by chance at all.
We can measure the size of nonresponse bias with one simple identity. Split the population into respondents and non-respondents. The true population proportion is a weighted average of the two groups:
Here \( p \) is the true population proportion (30% in the case). \( p_R \) is the proportion among respondents, which is what the survey reports. \( p_M \) is the proportion among the missing people, the non-respondents. \( r \) is the overall response rate, the share of selected people who answer. The right-hand form says the bias, \( p_R – p \), is the product of two things: how many people are missing, \( (1-r) \), and how different the missing people are from those who answered, \( (p_R – p_M) \). If either factor is zero, there is no bias. A low response rate alone does not guarantee bias, and a high response rate does not guarantee its absence. The same identity works for undercoverage if you read \( r \) as the share of the population that is on the frame and \( p_M \) as the proportion among the people who are left off.
Statistics Made Simple explains sampling bias and data collection with worked examples, diagrams and tables.Get the textbook · $12
Why this method fits the case
The gym question is a classic case for bias analysis for three reasons. First, the outcome (gym use) is closely linked to the decision to respond. People who use the gym are more interested in a gym survey. Whenever the thing you are measuring also drives who answers, nonresponse bias should be your first suspicion.
Second, we have a benchmark. The card-swipe records give the true figure, so we can calculate the bias exactly rather than guess at it. In real surveys you rarely know the truth, but you can often find a partial benchmark: a census figure, an administrative record or an earlier high-quality survey.
Third, the identity above turns a vague worry (“maybe the sample is unrepresentative”) into a number. It tells the sports office how far off they are and in which direction.
When is this framework not the right tool? If the error comes from the question itself, for example a leading question such as “Given how important fitness is, do you use the gym?”, that is response bias, not sampling bias. The right people answered; they just answered a slanted question, or gave the answer they thought sounded good. Likewise, if the sample is properly random and the estimate is off by a couple of points, that may simply be sampling variability, which a confidence interval is designed to describe. Bias analysis is for errors built into who ends up in the data.
Setting it up
Lay out every group in a table before calculating. It keeps the counts and percentages straight.
| Group | Population size | Response rate | Replies | Say “yes, weekly” |
|---|---|---|---|---|
| Gym users | 600 | 70% | 420 | 420 |
| Non-users | 1,400 | 45% | 630 | 0 |
| Total | 2,000 | 52.5% | 1,050 | 420 |
| Proportion “yes” | \( p = 600/2000 = 0.30 \) | \( r = 0.525 \) | \( p_R = 420/1050 = 0.40 \) | |
The 950 non-respondents are 180 users (600 − 420) and 770 non-users (1,400 − 630), so \( p_M = 180/950 \approx 0.189 \).
Solving it
Example 1 (nonresponse). Using the gym survey, find the survey estimate, the bias, and check the bias with the identity \( p_R – p = (1-r)(p_R – p_M) \).
The survey overstates weekly gym use by 10 percentage points because keen users were more likely to reply.
Example 2 (undercoverage). A city council wants to know what share of 1,200 households support a new bus route. It samples from a list of households with a registered landline. Only 750 households are on the list; 450 are not. Support is 30% among listed households and 55% among unlisted ones (who tend to be younger and more likely to use buses). Even if every listed household answers honestly, what will the survey report, and what is the bias?
Undercoverage makes the council underestimate support by about 9.4 points, and no amount of extra sampling from the same list would fix it.
Example 3 (voluntary response versus a random sample). A local news website runs an online poll: “Should the town ban cars from the high street?” Of 420 votes, 62% say “No”. The same week, a polling firm phones a simple random sample of 800 adults and finds 34% say “No”. A student computes a 95% margin of error for each. Which result should be trusted?
Trust the random sample. A margin of error on a voluntary-response poll is meaningless because it measures random error only, not the self-selection error.
Make spotting sampling bias automatic with the Practice Questions book: 1,569 questions, all fully worked.Get 1,569 questions · $9
What the answer tells us
The gym survey’s 40% is not a slightly noisy version of 30%. It is a confident answer to the wrong question. It describes “students who chose to reply to a gym email”, not “all undergraduates”. If the sports office used it to plan, it would over-equip the gym by a third.
Reasonableness check. The direction makes sense. The group with the higher response rate (users) also has the higher value of the outcome (100% “yes”), so it is over-represented and the estimate is pulled up. Whenever you identify a bias, ask “who is over-represented, and what do they tend to say?” That tells you the direction before any calculation.
Sensitivity. How bad the bias is depends on how unequal the response rates are. Figure 1 holds the gym users’ response rate at 70% and varies the non-users’ rate. If non-users reply at only 10%, the survey would report 75%. At 45% (the case) it reports 40%. Only when both groups reply at the same 70% does the estimate hit the true 30%. Notice that even a modest gap in response rates produces a large error when the outcome is closely linked to the decision to respond.
A famous real example. In 1936 the Literary Digest magazine mailed about 10 million ballots, using lists such as telephone directories and car registrations, and received almost 2.4 million back. It predicted that Alf Landon would beat Franklin Roosevelt. Roosevelt won with more than 60% of the popular vote[1]. For years the textbook story blamed undercoverage: in the Depression, people with phones and cars were better off and more likely to vote Republican. Later analysis of a 1937 Gallup survey suggested that nonresponse was at least as important, because Landon supporters were more likely to return the ballot[2]. Either way, 2.4 million replies did not save the poll. Size was never the problem.
What fixes bias? Better design, not more data. For undercoverage, improve the frame (combine lists, use address-based sampling). For nonresponse, follow up non-respondents, offer different ways to reply, keep the survey short, and compare respondents with known population figures. Weighting can reduce bias when you know which groups are under-represented, but it cannot fix differences you have not measured[1]. For voluntary response, the only real fix is to stop letting people pick themselves and use random selection instead[3].
To go further, our Statistics Made Simple textbook covers sampling frames, random sampling methods and the main sources of bias step by step, and the Statistics Made Simple Practice Questions book gives you hundreds of “identify the bias” questions with fully worked answers.
Master data collection and sampling with both Statistics Made Simple books for $16.Both books · $16
Practice questions
Try each question before opening the worked answer. The last one is written in exam style with a mark scheme.
EasyQ1. A radio station asks listeners to text in whether the new road layout is “better” or “worse”. 81% of 1,900 texts say “worse”. Name the type of bias and state its likely direction. Show worked answer
People who are unhappy with the change feel more strongly and are more likely to text.
So 81% probably overestimates the proportion of all drivers who think the layout is worse.
EasyQ2. To estimate average screen time of teenagers in a town, a researcher randomly selects 200 names from the membership list of the town’s sports clubs. What is the main problem? Show worked answer
Club members are likely to spend more time on sport and less on screens.
So the estimate will probably understate average screen time.
MediumQ3. A clinic emails all of its patients to ask how long they waited at their last visit. The response rate is 75%. Records show the mean waiting time was 40 minutes for patients who replied and 52 minutes for patients who did not. Find the true mean waiting time and the bias of the survey mean. Show worked answer
Bias \( = 40 – 43 = -3 \) minutes.
Check with the identity: \( (1-r)(\bar y_R – \bar y_M) = 0.25 \times (40 – 52) = -3 \).
The survey underestimates waiting time by 3 minutes, because patients who waited longest were less likely to reply.
MediumQ4. In a town of 1,000 adults, 200 are aged under 30. A survey contacts all adults; 60% of under-30s and 15% of over-30s reply. What proportion of the replies come from under-30s, and is this group over- or under-represented? Show worked answer
Under-30 share of replies \( = 120/240 = 0.50 \), compared with \( 200/1000 = 0.20 \) in the population.
Under-30s are heavily over-represented, so any outcome linked to age will be biased towards the views of younger adults.
ExamQ5. A school has 4,000 students; 30% walk to school. The head teacher wants the proportion who walk and posts a survey link in the school’s parent app. 70% of parents of walkers and 45% of parents of non-walkers respond. (a) State the type of bias. (b) Calculate the proportion of walkers among the responses. (c) Explain why sending the survey to more families in the same way would not fix the problem. [6 marks] Show worked answer
(b) Walker replies \( = 0.70 \times 1200 = 840 \); non-walker replies \( = 0.45 \times 2800 = 1260 \). [1]
Proportion \( = 840/(840+1260) = 840/2100 = 0.40 \). [1]
This overestimates the true 30% by 10 percentage points. [1]
(c) Bias is systematic: the same unequal response rates would apply to any extra families, so the estimate would stay near 40%. [1]
A larger sample only reduces random variability; to reduce bias the school must raise response among non-walkers (for example, follow-up or paper forms). [1]
Frequently asked questions
Is sampling bias the same as sampling error?
No. Sampling error (or sampling variability) is the random difference between a sample statistic and the population value that happens because we only looked at part of the population. It averages out over many samples and shrinks as the sample grows. Sampling bias is systematic: it pushes estimates in the same direction every time and does not shrink with sample size.
Does a low response rate always mean the results are biased?
Not always. The identity \( p_R – p = (1-r)(p_R – p_M) \) shows that bias needs both missing people and a difference between them and the respondents. If non-respondents happen to be just like respondents on the question asked, a low response rate does little harm. The danger is that you usually cannot check this, so a low response rate is a warning sign that needs investigating.
How is response bias different from nonresponse bias?
Nonresponse bias is about who answers. Response bias is about what people say once they answer: leading questions, social desirability (over-reporting good habits), poor memory or an interviewer’s influence. A survey can suffer from both at once.
Can random sampling remove all bias?
Random selection removes the researcher’s own selection bias and lets us measure random error. It cannot remove undercoverage if the frame is incomplete, nonresponse if selected people do not reply, or response bias from badly worded questions. Random sampling is necessary for an unbiased survey, but not sufficient.
References
- Lohr, S. L. and Brick, J. M. (2017) ‘Roosevelt predicted to win: revisiting the 1936 Literary Digest poll’, Statistics, Politics and Policy, 8(1), pp. 65–84. doi:10.1515/spp-2016-0006.
- Squire, P. (1988) ‘Why the 1936 Literary Digest poll failed’, Public Opinion Quarterly, 52(1), pp. 125–133. doi:10.1086/269085.
- Illowsky, B. and Dean, S. (2023) Introductory Statistics 2e. Houston, TX: OpenStax. https://openstax.org/details/books/introductory-statistics-2e
- Starnes, D. S. and Tabor, J. The Practice of Statistics. New York: W. H. Freeman.
- College Board. AP Statistics Course and Exam Description. New York: College Board.
