A magazine in 1936 mailed 10 million survey postcards asking Americans who they’d vote for in the presidential election, and predicted a landslide for Alf Landon. Franklin Roosevelt won in one of the largest landslides in American history. The magazine had 2.4 million responses — a huge sample — and got the result completely wrong. The problem wasn’t sample size. It was how the sample was chosen: postcards went to telephone owners and car registration holders, a group skewed wealthy in the middle of the Great Depression, and wealthier Americans leaned Republican. This is the single most important lesson in sampling: a large biased sample is worse than a small unbiased one, because it gives you false confidence in a wrong answer.
Population vs Sample: A Quick Recap
The population is the entire group you want to learn about; the sample is the subset you actually observe. Since surveying an entire population is usually impossible or impractical, the sampling method — the procedure used to select which population members end up in the sample — determines whether your sample’s statistics can be trusted to represent the population’s true parameters.
A sample is representative when it mirrors the population’s relevant characteristics. A sample is biased when its selection method systematically favours certain types of population members over others, producing a sample that misrepresents the population even before any data is collected.
Simple Random Sampling
In simple random sampling (SRS), every member of the population has an equal chance of being selected, and every possible sample of a given size is equally likely. In practice, this means numbering every population member and using a random number generator (or a random number table) to select the sample — no human judgement is involved in who gets chosen.
Advantage: SRS is the theoretical gold standard — it eliminates selection bias by construction, and the mathematics behind confidence intervals and hypothesis tests assumes SRS as the baseline. Disadvantage: it requires a complete list of the population (a “sampling frame”), which is often impossible to obtain in practice — you’d need a full list of every customer, every voter, every patient, which frequently doesn’t exist in a usable form.
Systematic Sampling
Systematic sampling selects every k-th member from an ordered list, after a random starting point. If you want a sample of 100 from a population of 5,000, you’d calculate k = 5000/100 = 50, pick a random starting point between 1 and 50, and then select every 50th person from there.
Advantage: much easier to execute than SRS in practice — no need to generate hundreds of random numbers, just count down a list. It also spreads the sample evenly across the population. Disadvantage: if the list has a hidden periodic pattern that happens to align with your interval k, the sample can become badly biased. A classic example: sampling every 10th house on a street where every 10th house happens to be a corner lot with systematically different characteristics.
Stratified Sampling
Stratified sampling divides the population into non-overlapping subgroups (“strata”) based on a shared characteristic — age group, income bracket, region — then takes a random sample from within each stratum, usually proportional to that stratum’s share of the population.
Worked example: A university has 3,000 undergraduates (60%), 1,500 postgraduates (30%), and 500 staff (10%), and wants a stratified sample of 200 for a satisfaction survey, proportional to these groups.
Undergraduates: 200 × 0.60 = 120
Postgraduates: 200 × 0.30 = 60
Staff: 200 × 0.10 = 20
Within each stratum, individuals are then selected using simple random sampling. This guarantees the sample’s composition matches the population’s composition exactly on the stratifying variable — something SRS alone can’t guarantee, since a pure random draw could by chance under-represent a smaller group.
Advantage: generally produces more precise estimates than SRS of the same size, especially when the stratifying variable is related to the outcome being measured, because it removes between-stratum variation as a source of sampling error. Disadvantage: requires knowing the stratum each population member belongs to in advance, which adds complexity to the sampling frame.
Cluster Sampling
Cluster sampling divides the population into groups (“clusters”) that ideally each mirror the population’s diversity — often based on geography, like city blocks or schools — then randomly selects entire clusters, and every member within a selected cluster is included (or a further random sample is drawn within each selected cluster, called two-stage cluster sampling).
This is the technique’s key distinction from stratified sampling, and the one students confuse most often: stratified sampling samples from within every stratum; cluster sampling samples entire clusters and ignores the rest. A national survey randomly selecting 50 out of 3,000 school districts, then surveying every student in those 50 districts, is cluster sampling — the districts not selected contribute nothing to the sample.
Advantage: dramatically cheaper and more practical when the population is spread over a large area, since it avoids the cost of surveying scattered individuals across the entire region — you only need to physically visit the selected clusters. Disadvantage: less statistically efficient than stratified sampling for a given sample size, because members within a cluster tend to be similar to each other (neighbours often share income levels, for instance), so a cluster contributes less unique information than the same number of individuals spread randomly across the population.
Comparing the Four Methods
| Method | Needs Full Population List? | Cost | Precision |
|---|---|---|---|
| Simple Random | Yes | High | Good (baseline) |
| Systematic | Yes (ordered) | Medium | Good, unless periodic bias |
| Stratified | Yes, plus stratum labels | Medium-High | Best |
| Cluster | No, just cluster list | Low | Lower than the others |
Sampling Bias vs Non-Response Bias
Sampling bias occurs when the method used to select the sample systematically excludes or under-represents part of the population — the 1936 magazine poll is the textbook case. Non-response bias is a different, equally dangerous problem: even with a perfectly designed sample, if the people who don’t respond differ systematically from those who do, the responses you actually collect are skewed. A customer satisfaction survey where only very angry or very happy customers bother to respond suffers from non-response bias even if the initial sample was drawn with pure SRS.
Both problems can coexist, and neither is fixed by increasing sample size — a larger biased sample just estimates the wrong thing more precisely. This is why survey response rates, and the demographics of who does and doesn’t respond, matter as much as the sampling method itself.
Real-World Applications
National statistical agencies use stratified sampling extensively — a labour force survey might stratify by region and industry to ensure every economically important subgroup is adequately represented, since a pure SRS of a country’s workforce could easily under-sample small but important sectors. Political polling organisations often use multi-stage cluster sampling: selecting a random sample of voting districts, then a random sample of households within each, since surveying scattered individuals across an entire country is logistically prohibitive. Quality-control auditors in manufacturing commonly use systematic sampling — inspecting every 20th unit off a production line — because it’s simple to implement on a moving line and spreads inspection evenly across a shift.
Common Mistakes
Confusing stratified and cluster sampling. This is the single most common error. Remember: stratified samples from within every group; cluster samples entire groups and leaves others out completely.
Assuming a large sample fixes a bad sampling method. As the 1936 election poll demonstrates, sample size cannot correct for a systematically biased selection process — it only makes you more confident in the wrong answer.
Ignoring non-response as a source of bias. Students often evaluate whether a sampling method is theoretically sound but forget to ask who actually responded, and whether responders differ meaningfully from non-responders.
Practice Question
A school wants to survey students about cafeteria food. They randomly select 5 out of the school’s 30 homeroom classes and survey every student in those 5 classes. Identify the sampling method used and state one advantage and one disadvantage.
Answer: This is cluster sampling — entire homeroom classes (clusters) are randomly selected, and every member of a selected cluster is included, rather than sampling individuals from across all 30 classes.
Advantage: much more practical to administer — the survey team only needs to visit 5 classrooms rather than pulling scattered individual students out of all 30 classes, saving significant time and disruption.
Disadvantage: if homeroom classes are not similarly diverse (for example, if classes are grouped by academic ability or friend groups with similar tastes), the selected 5 classes may not represent the full range of opinions across the school, reducing the precision of the results compared to a method that sampled more broadly.
References
1. Moore, D.S., McCabe, G.P. and Craig, B.A. (2021) Introduction to the Practice of Statistics. W.H. Freeman.
2. Groves, R.M. et al. (2009) Survey Methodology. Wiley.
3. Triola, M.F. (2022) Elementary Statistics. Pearson.
4. Squire, P. (1988) ‘Why the 1936 Literary Digest Poll Failed’, Public Opinion Quarterly, 52(1), pp. 125–133.
5. Levy, P.S. and Lemeshow, S. (2013) Sampling of Populations: Methods and Applications. Wiley.