- A well-designed experiment rests on four principles: control (a comparison group and fixed conditions), randomisation (chance decides who gets which treatment), replication (enough units per treatment) and blocking (grouping similar units first).
- In our wheat trial, 12 plots on a slope are split into 4 blocks of 3. Within each block, chance assigns a control, fertiliser A and fertiliser B.
- Answer to the case: fertiliser A adds about 0.8 t/ha and fertiliser B about 1.15 t/ha over the control, and B beats A in every one of the 4 blocks.
- Blocking removes the slope from the comparison: the spread of the A-minus-control differences (SD 0.08 t/ha) is about nine times smaller than the spread of yields within a treatment (SD about 0.75 t/ha).
- Exam point: when asked to “describe a design”, say how many units, how they are grouped, how chance is used (in detail), and what is measured.
The case: which fertiliser gives more wheat?
A farm cooperative wants to know whether two new fertilisers, A and B, raise wheat yields enough to be worth buying. It has one field to test them on. The field is divided into 12 plots of equal size. There is a catch: the field lies on a gentle slope. The soil at the bottom is heavier and often waterlogged, and last season’s yields were highest at the top of the slope and lowest at the bottom.
The agronomist could simply put fertiliser A on the top four plots, B on the middle four and nothing on the bottom four. That would be quick. It would also be useless, because any difference in yield could be caused by the slope rather than the fertiliser. We would have no way to tell the two apart.
Instead, the agronomist designs an experiment. She groups the plots into 4 horizontal strips (blocks) of 3 plots each, so that the plots within a strip sit at the same height on the slope. Inside each strip, she uses chance to decide which plot gets the control (no fertiliser), which gets A and which gets B. At harvest she records the yield of every plot in tonnes per hectare (t/ha). This post explains each design choice, then works out what the data say. (The figures are illustrative.)
The concept: the four principles of experimental design
An experiment is a study in which the researcher deliberately imposes a treatment on the experimental units and measures the response. That is what separates it from an observational study, where we only watch what happens. Because the researcher controls who gets which treatment, a well-designed experiment can give strong evidence of cause and effect. A badly designed one gives evidence of nothing.
The vocabulary matters. The experimental units are the things treatments are applied to (here, plots; with people they are called subjects or participants). The factor is the explanatory variable the researcher sets (fertiliser type). Its values are the levels, and each combination applied to a unit is a treatment (control, A, B). The response variable is what we measure (yield).
Control means two things. First, include a comparison group, often a control group that receives no treatment or a placebo, so that we know what would have happened anyway. Second, keep other conditions the same for every unit: the same wheat variety, sowing date, seed rate and harvesting method. Anything you cannot hold fixed becomes a possible confounding variable.
Randomisation means using a chance process to assign treatments to units. It does not make the groups identical, but it makes them alike on average, including on variables nobody thought of. It also stops the researcher, consciously or not, from giving the best plots to the favoured treatment.
Replication means applying each treatment to several units. Natural variation between plots is unavoidable. With only one plot per treatment, a lucky plot could make a useless fertiliser look good. With several, chance differences tend to average out, and we can measure how big the natural variation is.
Blocking means grouping units that are similar on a variable expected to affect the response, then randomising separately within each block. It is the experimental version of stratified sampling. Here the slope is the blocking variable.
Replication is what makes the comparison precise. If plot yields within a treatment vary with standard deviation \( \sigma \), and each treatment is applied to \( n \) plots, then the standard error of the difference between two treatment means is
Here \( \bar y_1 \) and \( \bar y_2 \) are the mean responses for two treatments, \( \sigma \) is the standard deviation of the response between units given the same treatment (the “noise”), and \( n \) is the number of replicates per treatment. The formula shows the two levers a designer has. Replication increases \( n \), which shrinks the standard error with \( \sqrt{n} \): four times as many plots halves it. Blocking attacks \( \sigma \) itself: if we compare treatments within blocks, the relevant noise is only the variation inside a block, which is much smaller than the variation across the whole slope.
Statistics Made Simple explains experimental design with worked examples, diagrams and tables.Get the textbook · $12
Why this method fits the case
The cooperative’s question is causal: does the fertiliser raise yield? Only an experiment can answer that cleanly. If they instead compared farms that happen to use fertiliser A with farms that do not, the A farms might also have better soil, newer machinery or more experienced managers. That is an observational study, and confounding would be everywhere.
A randomised block design suits this field in particular because we know in advance about one big source of variation, the slope. Ignoring it would not bias a completely randomised design, because randomisation still balances the slope on average. But with only 12 plots, a completely randomised design could easily, by bad luck, give fertiliser B three of the top plots. Even when it does not, the slope adds a lot of noise to every comparison. Blocking guarantees that each treatment appears exactly once at each height, and lets us remove the slope effect from the analysis.
When is blocking not the right tool? If you have no variable that you expect to affect the response, blocking adds complexity for no gain; a completely randomised design is simpler and fine. If the blocks are formed on a variable that has nothing to do with the response, you lose a little precision. And if the number of units in each block is smaller than the number of treatments, a simple randomised block design does not work; you would need a more advanced (incomplete block) design. Finally, design cannot rescue an experiment where the treatment cannot ethically or practically be imposed. You cannot randomly assign people to smoke, for example; then a carefully analysed observational study is the best you can do.
Setting it up
Here are the yields recorded at harvest, in tonnes per hectare.
| Block (height on slope) | Control | Fertiliser A | Fertiliser B | Block mean |
|---|---|---|---|---|
| 1 (top) | 6.2 | 7.0 | 7.4 | 6.87 |
| 2 | 5.6 | 6.5 | 6.7 | 6.27 |
| 3 | 5.1 | 5.8 | 6.3 | 5.73 |
| 4 (bottom) | 4.5 | 5.3 | 5.6 | 5.13 |
| Treatment mean | 5.35 | 6.15 | 6.50 | 6.00 overall |
| SD within treatment | 0.72 | 0.75 | 0.75 |
The block means fall from 6.87 to 5.13 t/ha down the slope, a range of 1.73 t/ha. That is larger than any of the treatment effects, which is exactly why blocking was worth doing.
Solving it
Example 1 (randomising within blocks). Describe how to randomly assign the three treatments within block 1, in enough detail that someone else could carry it out.
Each block gets every treatment exactly once, and chance alone decides which plot within a block gets which treatment.
Example 2 (estimating treatment effects within blocks). Use the table to estimate the effect of each fertiliser compared with the control, and show why the within-block comparison is more precise than comparing raw treatment groups.
Fertiliser A adds about 0.8 t/ha (15% over the control mean of 5.35) and B about 1.15 t/ha (21%), and blocking makes those estimates far more precise.
Example 3 (how much replication?). Suppose a trial on a different, flatter field uses a completely randomised design, and the plot-to-plot SD there is \( \sigma = 0.4 \) t/ha. How many plots per treatment are needed for the SE of a difference in means to be at most 0.15 t/ha?
Without blocking, precision has to be bought with many more plots; blocking is often the cheaper route.
Make experimental design automatic with the Practice Questions book: 1,569 questions, all fully worked.Get 1,569 questions · $9
What the answer tells us
Both fertilisers raise yield, and B raises it more. The B-versus-A advantage of 0.35 t/ha is smaller than either effect against the control, but it is consistent: B beat A in all four blocks. With a sign pattern like that in such a tightly controlled trial, chance is an unlikely explanation. A formal analysis (for example, analysis of variance for a randomised block design) would confirm it[2].
Reasonableness check. Figure 1 shows the yields block by block. Yields fall steadily from the top of the slope to the bottom, for every treatment. But within every block the order never changes: control lowest, then A, then B. If the treatment effects had varied wildly from block to block, we would worry about an interaction (for example, a fertiliser that works only on wet soil). They do not, so a single average effect is a fair summary.
Sensitivity. Imagine the agronomist had not blocked and, by bad luck, randomisation had placed all four control plots at the top and all four B plots at the bottom. The raw comparison could then show the control beating fertiliser B, because the slope effect (a 1.73 t/ha range in block means) is bigger than the fertiliser effect. Blocking removes that risk completely.
Scope of the conclusion. The trial used one field, one wheat variety and one season. The cause-and-effect conclusion is strong for conditions like these. Extending it to other soils, climates or crops needs either more trials or careful judgement. That is the difference between internal validity (the design is sound) and external validity (the result generalises). The modern ideas of randomisation, replication and blocking were set out by R. A. Fisher, who developed them for exactly this kind of agricultural field trial[1].
For a full treatment of experiments versus observational studies, completely randomised and block designs and matched pairs, see our Statistics Made Simple textbook, and practise writing randomisation procedures with the Statistics Made Simple Practice Questions book.
Master study design and data collection with both Statistics Made Simple books for $16.Both books · $16
Practice questions
Try each question before opening the worked answer. The last one is written in exam style with a mark scheme.
EasyQ1. A researcher tests whether a new toothpaste reduces cavities. 200 children are randomly split into two groups; one uses the new toothpaste and one an ordinary toothpaste. Identify the experimental units, the factor, the treatments and the response variable. Show worked answer
Factor: type of toothpaste.
Treatments: new toothpaste; ordinary toothpaste.
Response variable: number of new cavities over the study period.
EasyQ2. In the wheat trial, why was a control group (no fertiliser) included instead of just comparing A with B? Show worked answer
The control shows what yield would be without fertiliser, so we can estimate each fertiliser’s actual gain (0.80 and 1.15 t/ha).
MediumQ3. A within-block experiment gives these B-minus-A differences in five blocks: 0.4, 0.1, 0.3, 0.5, 0.2 t/ha. Find the mean difference and its standard error. Show worked answer
Deviations: 0.1, −0.2, 0, 0.2, −0.1; squares sum to \( 0.01+0.04+0+0.04+0.01 = 0.10 \).
SD \( = \sqrt{0.10/4} = \sqrt{0.025} \approx 0.158 \).
SE \( = 0.158/\sqrt5 \approx 0.071 \) t/ha.
MediumQ4. Plot-to-plot SD is 0.6 t/ha. A trial uses 8 plots per treatment in a completely randomised design. Find the SE of the difference between two treatment means. How many plots per treatment would halve it? Show worked answer
Halving SE needs \( \sqrt{n} \) to double, so \( n \) must be multiplied by 4.
\( n = 32 \) plots per treatment; check \( 0.6\sqrt{2/32} = 0.6 \times 0.25 = 0.15 \).
ExamQ5. A gym wants to compare two 8-week training programmes for improving 5 km run times. 40 volunteers are available: 20 are experienced runners and 20 are beginners. (a) Explain why blocking by experience is sensible. (b) Describe how to assign programmes using a randomised block design. (c) Explain why the result may not apply to all gym members. [6 marks] Show worked answer
(b) Within the experienced block, label runners 1–20 and use a random number generator to choose 10 distinct labels for programme 1; the other 10 get programme 2. [1] Repeat separately for the 20 beginners. [1] After 8 weeks, compare improvement in 5 km time between programmes within each block. [1]
(c) The participants were volunteers, not a random sample of members, so they may be more motivated; conclusions about cause apply to people like the volunteers. [1]
Frequently asked questions
What is the difference between blocking and stratified sampling?
They use the same idea, grouping similar units first, but for different purposes. Stratified sampling chooses who is in the sample so that it represents the population. Blocking decides how treatments are assigned within an experiment so that comparisons are more precise. One is about selection, the other about assignment.
Is a matched-pairs design a kind of block design?
Yes. A matched-pairs design is a randomised block design with blocks of size two: either two similar units, or one unit that receives both treatments in a random order. The paired t-test is the analysis that goes with it.
Why does randomisation allow us to claim cause and effect?
Because chance, not the researcher or the units, decides who gets each treatment, the treatment groups differ only by chance on every other variable. If the response differs by more than chance alone would plausibly produce, the treatment is the only systematic difference left, so it is the most reasonable cause.
Do experiments always need a placebo?
No. A placebo is needed when the subjects’ expectations could affect the response, which is common in medicine and psychology. Wheat plots have no expectations, so a no-fertiliser control is enough. When people are involved, blinding (subjects and, ideally, evaluators not knowing who got which treatment) is also important.
References
- Fisher, R. A. (1935) The Design of Experiments. Edinburgh: Oliver and Boyd.
- NIST/SEMATECH. e-Handbook of Statistical Methods, Process Improvement (design of experiments). https://www.itl.nist.gov/div898/handbook/
- Illowsky, B. and Dean, S. (2023) Introductory Statistics 2e. Houston, TX: OpenStax. https://openstax.org/details/books/introductory-statistics-2e
- Moore, D. S., McCabe, G. P. and Craig, B. A. Introduction to the Practice of Statistics. New York: W. H. Freeman.
- College Board. AP Statistics Course and Exam Description. New York: College Board.
