Statistics guide, Unit 1, step 3 of 22Previous step Next step

Step 3: Experimental Design

By the EconTutorials team. 11-minute read with 6 practice questions.

SyllabusAP Stats 1.13

In 30 seconds

An observational study watches; an experiment imposes a treatment and measures the response.
A well-designed experiment uses four principles: comparison, random assignment, control and replication.
Completely randomised, randomised block and matched pairs designs are the three designs you must be able to describe.
Random sampling lets you generalise to a population; random assignment lets you claim cause and effect. Exam answers must name which one you have.

Video lesson coming soonA short animated lesson for this step will appear here.

Watching versus doing: observational studies and experiments

Suppose students who drink coffee in the morning seem to get higher marks. If you ask 200 students how much coffee they drink and compare their marks, you have changed nothing; you have recorded what was already happening. That is an observational study. If instead you decide who drinks coffee before a test and who drinks water, then measure the marks, you have imposed a treatment. That is an experiment.

Coffee drinkers might also sleep less, which could explain the marks. An observational study shows association; only a well-designed experiment shows cause and effect.

Key termObservational study: a study that observes individuals and measures variables of interest but does not attempt to influence the responses.

Key termExperiment: a study that deliberately imposes treatments on individuals in order to measure their responses.

Confounding

Key termConfounding: when two variables are associated in such a way that their effects on a response variable cannot be distinguished from each other.

Here, sleep is linked to coffee drinking and to marks, so its effect cannot be separated from coffee’s.

The language of experiments

Experimental units: the objects the treatments are given to. When they are people, they are called subjects.
Factor: an explanatory variable that the experimenter controls, for example “type of drink” or “amount of fertiliser”.
Levels: the different values of a factor, for example “coffee” and “water”, or 0 g, 50 g and 100 g.
Treatment: a specific condition applied to the units. With one factor, each level is a treatment. With two factors, each combination of levels is a treatment: 2 drinks × 3 amounts of sleep gives 6 treatments.
Response variable: the outcome you measure, for example test score.

The four principles of a good experiment

Comparison. Use at least two treatment groups so you can compare them. Often one is a control group that receives no treatment, an existing treatment or a placebo. Without it, you cannot separate the treatment’s effect from anything else that changed.
Random assignment. Use a chance process to decide which units get which treatment. This tends to create roughly equivalent groups, even for variables you never thought of, so the only systematic difference is the treatment.
Control. Keep other variables that might affect the response the same for all groups (same test, room, time, instructions). This reduces variability and prevents confounding.
Replication. Give each treatment to enough units that chance differences between groups are likely to be small.

Key termStatistically significant: a difference between groups too large to be plausibly explained by the chance variation in random assignment.

Placebos and blinding

People often feel better simply because they believe they are being treated: the placebo effect. A placebo is a fake treatment that looks like the real one, such as a sugar pill. If the control group gets a placebo, both groups experience “being treated”, so any difference can be put down to the treatment itself.

Expectations also affect whoever measures the response, so experiments use blinding:

Single-blind: either the subjects or the people measuring the response do not know which treatment each subject received.
Double-blind: neither the subjects nor the people who interact with them and measure the response know which treatment each subject received.

Common mistakeMixing up random sampling (choosing who is in the study) with random assignment (deciding which treatment each unit gets). An experiment on volunteers can use random assignment with no random sampling.

Three experimental designs

1. Completely randomised design

In a completely randomised design, every experimental unit is allocated to a treatment purely by chance. With 40 subjects and 2 treatments, you would randomly split them into two groups of 20.

To describe it for full credit: label the subjects 1 to 40, use a random number generator to pick 20 different labels (ignoring repeats), give those 20 Treatment A and the rest Treatment B.

DiagramSketch a flow chart from left to right: “40 students” → arrow labelled “Random assignment” splitting into “Group 1: 20 students → revise with music” and “Group 2: 20 students → revise in silence” → both branches join at “Compare mean quiz scores”.

2. Randomised block design

Sometimes you know in advance that a variable, such as prior grades, will strongly affect the response. In a randomised block design, you first split the units into blocks of similar units, then randomly assign treatments separately within each block.

Key termBlock: a group of experimental units that are known before the experiment to be similar in some way that is expected to affect the response to the treatments.

Why block? Blocking accounts for the variation caused by the blocking variable, so it is easier to see the effect of the treatment. Remember the slogan: “Block what you can; randomise what you cannot.” Blocking is to experiments what stratifying is to samples.

DiagramStart with “40 students” and split them (no randomness yet) into “Block 1: 16 high achievers” and “Block 2: 24 other students”. From each block, an arrow labelled “Random assignment” splits into music and silence groups (8 and 8 in Block 1; 12 and 12 in Block 2), ending in “Compare quiz scores” within each block.

3. Matched pairs design

A matched pairs design is a special block design where each block has just two units (or one unit used twice), and there are two treatments.

Pairs of similar units: pair students with similar previous grades. Within each pair, toss a coin: heads means the first student gets music, the other gets silence.
Each unit gets both treatments: each student revises once with music and once in silence. The order is decided at random (for example by a coin toss), so that any “practice effect” from doing a second quiz does not always favour the same treatment.

You then analyse the differences within each pair. This removes much of the variation between people.

Scope of inference: what can you conclude?

How the data were produced decides what you may conclude:

Was there random sampling from a population? If yes, you can generalise the results to that population.
Was there random assignment to treatments? If yes, you can conclude that the treatment caused the difference in response.

	Random assignment: yes	Random assignment: no
Random sampling: yes	Can conclude cause and effect and generalise to the population.	Can generalise to the population, but cannot conclude cause and effect (association only).
Random sampling: no	Can conclude cause and effect, but only for subjects like those in the study.	Cannot conclude cause and effect and cannot generalise; results apply only to the individuals studied.

Worked example: designing an experiment, AP style

Worked example

A teacher wants to know whether listening to instrumental music while revising improves scores on a 20-question vocabulary quiz. 40 students in her Year 11 class volunteer. (a) Identify the experimental units, treatments and response variable. (b) Describe a completely randomised design. (c) Explain why random assignment is important here. (d) The teacher knows that 16 of the students already score highly on vocabulary. Describe how she could use blocking, and explain the benefit.

Step 1Units: the 40 volunteer students. Treatments: revising with instrumental music, revising in silence. Response: score out of 20 on the vocabulary quiz.

Step 2Label the students 1 to 40. Use a random number generator to select 20 different labels from 1 to 40, ignoring repeats. These 20 revise a set list of 30 words for 20 minutes with instrumental music through headphones; the other 20 revise the same list for 20 minutes wearing silent headphones. All revise in the same room at the same time, then take the same quiz. Compare the mean quiz scores of the two groups.

Step 3Random assignment should create groups roughly equivalent in ability, motivation and sleep. Then, if the music group scores significantly higher, the teacher can attribute the difference to the music rather than to differences between the groups.

Step 4Form two blocks: the 16 high scorers and the other 24. Within each block, randomly assign half to music and half to silence (8 and 8; 12 and 12), and compare within each block. Because prior ability affects quiz scores, blocking removes that variation, making any effect of music easier to detect.

Scope: with random assignment, a significant difference would show music causes a change in scores, but only for students like these 40 volunteers, who were not a random sample.

Exam tipAP design FRQs reward answers that (1) describe random assignment so someone else could repeat it (labels, method, group sizes, repeats), (2) say what each group does, (3) name the response and how groups are compared, and (4) explain why in context. “Randomisation removes bias” earns little; “random assignment balances prior ability between the music and silence groups, so a difference in scores can be attributed to the music” earns the mark. For confounding, link the variable to both the explanatory variable and the response.

Practise: 6 questions

Try each one on paper first, then check the worked answer.

EasyQ1. A researcher asks 500 adults how many hours of TV they watch and measures their blood pressure. Is this an observational study or an experiment? Explain. Show worked answer

Observational study. The adults chose their own TV hours; no treatment was imposed. It can show association, not that TV causes higher blood pressure.

EasyQ2. A farmer tests three amounts of fertiliser (0 kg, 5 kg, 10 kg) on two varieties of wheat, using 36 plots. Identify the factors, levels, number of treatments and experimental units. Show worked answer

Factors: amount of fertiliser and wheat variety. Levels: fertiliser has 3 levels (0, 5, 10 kg); variety has 2 levels. Treatments: 3 × 2 = 6 combinations. Experimental units: the 36 plots of land (6 plots per treatment).

MediumQ3. An observational study finds that students who eat breakfast get higher exam marks. Name a possible confounding variable and explain how it creates confounding. Show worked answer

Amount of sleep. Well-rested students may be more likely to have time for breakfast, and well-rested students also tend to do better in exams, so we cannot tell whether breakfast or sleep explains the higher marks.

MediumQ4. In a trial of a new pain-relief tablet, why should the control group receive a placebo, and why should the trial be double-blind? Show worked answer

Placebo: patients may report less pain just because they think they are treated. With identical-looking tablets, this placebo effect is the same in both groups, so any extra improvement can be credited to the drug. Double-blind: patients who knew their group might report differently, and doctors who knew might (even unconsciously) rate treated patients as better. Keeping both unaware prevents this bias.

MediumQ5. A researcher wants to compare two brands of running shoe for wear. She has 20 runners. Describe a matched pairs design and explain why it is better than giving 10 runners Brand A and 10 runners Brand B. Show worked answer

Each runner wears one shoe of each brand. For each runner, toss a coin: heads means Brand A goes on the left foot, tails the right. After a month, measure the wear on each shoe and analyse the difference (A − B) for each runner. Runners differ greatly in weight, distance and style. Comparing both brands on the same runner removes this variation, so a difference between brands is easier to detect.

ExamQ6. A company wants to know if a new app reduces how much time people spend on social media. It randomly selects 200 of its 5,000 employees and then randomly assigns 100 to use the app for a month; the other 100 do not. Average daily social media time is lower in the app group, and the difference is statistically significant. (a) Can the company conclude the app causes a reduction? (b) To whom can the result be generalised? Explain. [4 marks] Show worked answer

(a) Yes [1]. The employees were randomly assigned to use the app or not, so the groups should be roughly equivalent apart from the app; a statistically significant difference can therefore be attributed to the app [1]. (b) To all 5,000 employees of the company [1], because the 200 subjects were a random sample from them. It cannot be generalised beyond this company (for example to all adults), since other people were not sampled [1].

Check yourself

1. What is the main purpose of random assignment in an experiment?

It balances other variables between groups, allowing cause-and-effect conclusions.

2. Subjects are split by age group, then treatments are randomly assigned within each age group. This is:

Grouping first, then randomising within groups, is blocking.

3. A study uses volunteers and random assignment. What can you conclude?

Random assignment gives causation; without random sampling, you cannot generalise beyond similar subjects.

Your score: 0 out of 3

The short version

Observational studies show association; only well-designed experiments show cause and effect.
Know the vocabulary: experimental units (subjects), factors, levels, treatments, response variable.
Four principles: comparison, random assignment, control and replication. Use placebos and blinding when expectations could affect results.
Designs: completely randomised, randomised block (“block what you can, randomise what you cannot”) and matched pairs.
Scope of inference: random sampling lets you generalise; random assignment lets you claim causation.