Two-Proportion z-Test: Comparing Two Percentages Step by Step, with an A/B Test

In 30 seconds

A two-proportion z-test asks whether the difference between two sample percentages is bigger than chance alone would usually produce.
Under the null hypothesis the two groups share one proportion, so you estimate it by pooling both samples.
The test statistic is the observed difference divided by its standard error; compare it with the standard normal distribution.
In our A/B test, the new checkout converted 6.5% of visitors against 5.0% for the old one: $z \approx 2.23$, two-sided $p \approx 0.026$, so the improvement is statistically significant at the 5% level.
Exam point: use the pooled proportion for the test, but the unpooled standard error for the confidence interval, and always check the large-counts condition.

The case: did the new checkout page really sell more?

An online shop that sells revision guides wants to know whether a redesigned checkout page turns more visitors into buyers. The team runs an A/B test for two weeks. Every visitor who reaches the basket is randomly sent to either the old page (version A) or the new page (version B). Nobody chooses which version they see.

At the end of the test the numbers are:

Version A (old page): 2,400 visitors, 120 purchases, a conversion rate of 5.0%.
Version B (new page): 2,400 visitors, 156 purchases, a conversion rate of 6.5%.

The new page looks better by 1.5 percentage points. But conversion is noisy. If you ran the old page twice on two random groups of 2,400 people, you would not get exactly 120 sales both times. So the real question is: is a gap of 1.5 points larger than random variation would normally produce? If it is, the shop can switch to the new page with some confidence. If it is not, the “win” may just be luck. (The figures are illustrative.)

This is exactly the job of the two-proportion z-test. The same method is used to compare recovery rates under two treatments, pass rates in two schools, vote shares in two regions, or defect rates on two production lines.

The concept: comparing two proportions with a z statistic

A proportion is a count divided by a sample size: the share of people with some characteristic. When we take a random sample, the sample proportion $\hat p$ varies from sample to sample. For large samples its sampling distribution is approximately normal, centred on the true proportion $p$, with standard deviation $\sqrt{p(1-p)/n}$.

When we have two independent samples, the difference $\hat p_1 – \hat p_2$ also has an approximately normal sampling distribution. Its centre is the true difference $p_1 – p_2$. Because the two samples are independent, the variances add, so its standard deviation is $\sqrt{p_1(1-p_1)/n_1 + p_2(1-p_2)/n_2}$.^[1]

The test starts by assuming there is no difference: $H_0: p_1 = p_2$. If that is true, both groups share one common proportion, and our best estimate of it comes from combining the two samples. This combined estimate is called the pooled proportion. We then ask how many standard errors the observed difference sits away from zero.

Key termTwo-proportion z-test: a significance test of $H_0: p_1 = p_2$ for two independent random samples or two randomly assigned groups. It uses the pooled sample proportion to compute the standard error of $\hat p_1 – \hat p_2$, and refers the resulting z statistic to the standard normal distribution.

\[ \hat p_c = \frac{x_1 + x_2}{n_1 + n_2}, \qquad z = \frac{\hat p_1 – \hat p_2}{\sqrt{\hat p_c(1-\hat p_c)\left(\dfrac{1}{n_1} + \dfrac{1}{n_2}\right)}} \]

Here $x_1$ and $x_2$ are the numbers of “successes” (purchases, recoveries, passes) in each group, and $n_1$ and $n_2$ are the group sizes. The sample proportions are $\hat p_1 = x_1/n_1$ and $\hat p_2 = x_2/n_2$. The pooled (combined) proportion $\hat p_c$ is the total number of successes divided by the total number of observations. The denominator is the standard error of the difference when the null hypothesis is true. The result $z$ tells you how many standard errors the observed difference lies from zero. A large $|z|$ means the data would be surprising if the two true proportions were equal.^[2]

Statistics Made Simple explains comparing two proportions with worked examples, diagrams and tables.Get the textbook · $12

The companion tool is the confidence interval for $p_1 – p_2$. It answers a different question: not “is there a difference?” but “how big is the difference likely to be?” Because the interval does not assume the null hypothesis, it uses each group’s own proportion in the standard error rather than the pooled one:

\[ (\hat p_1 – \hat p_2) \pm z^{*}\sqrt{\frac{\hat p_1(1-\hat p_1)}{n_1} + \frac{\hat p_2(1-\hat p_2)}{n_2}} \]

For 95% confidence, $z^{*} = 1.96$. For 90% it is 1.645 and for 99% it is 2.576.

Why this method fits the case

The A/B test has four features that point straight to the two-proportion z-test:

The response is categorical with two outcomes. Each visitor either buys or does not buy. We are comparing percentages, not averages of a measured quantity.
There are two separate groups. Version A visitors and version B visitors are different people. Nobody appears in both groups.
The groups were formed by random assignment. Randomisation makes the groups comparable and is what allows a cause-and-effect conclusion: if the difference is significant, the page caused it.
The samples are large. With 120 and 156 purchases, and over 2,000 non-purchases in each group, the normal approximation is very good.

It is not the right tool in several common situations:

If the response is a measurement (time on page, spend in pounds), compare means with a two-sample t-test instead.
If the same people are measured twice (for example, the share who pass a quiz before and after a lesson), the samples are paired, not independent. A matched-pairs method such as McNemar’s test is needed.
If you have more than two groups, or more than two outcome categories, use a chi-square test for homogeneity.
If the counts are small (say 3 successes out of 25), the normal approximation breaks down. An exact method such as Fisher’s exact test is safer.^[5]

Setting it up

Write down every quantity before you calculate anything. It keeps the arithmetic tidy and shows the examiner you know what each symbol means.

Quantity	Symbol	Version A (old)	Version B (new)
Visitors	$n$	2,400	2,400
Purchases	$x$	120	156
Sample proportion	$\hat p = x/n$	0.0500	0.0650
Non-purchases	$n – x$	2,280	2,244
Pooled proportion	$\hat p_c$	276 / 4,800 = 0.0575
Significance level	$\alpha$	0.05

Let $p_A$ be the true conversion rate with the old page and $p_B$ the true rate with the new page. The hypotheses are

\[ H_0: p_B – p_A = 0 \qquad H_a: p_B – p_A \neq 0 \]

We use a two-sided alternative because, before the test, the team did not know whether the redesign would help or hurt. Choosing a one-sided alternative after seeing the data would be cheating.

Conditions. (1) Random: visitors were randomly assigned to the two versions. (2) Independence: one visitor’s purchase decision does not affect another’s, and the groups are separate. (3) Large counts: the expected numbers of successes and failures under $H_0$ are $2400 \times 0.0575 = 138$ successes and $2400 \times 0.9425 = 2262$ failures in each group, all comfortably above 10.^[3]

Solving it

Worked example

Carry out the two-proportion z-test for the A/B test at the 5% significance level.

Step 1 Sample proportions: $\hat p_A = 120/2400 = 0.050$ and $\hat p_B = 156/2400 = 0.065$. Observed difference $\hat p_B – \hat p_A = 0.015$.

Step 2 Pooled proportion: $\hat p_c = (120 + 156)/(2400 + 2400) = 276/4800 = 0.0575$.

Step 3 Standard error under $H_0$: $\sqrt{0.0575 \times 0.9425 \times (1/2400 + 1/2400)} = \sqrt{0.05419 \times 0.000833} = \sqrt{0.00004516} \approx 0.006720$.

Step 4 Test statistic: $z = 0.015 / 0.006720 \approx 2.23$.

Step 5 P-value (two-sided): $2 \times P(Z > 2.23) \approx 2 \times 0.0128 = 0.0256$.

Step 6 Decision: $0.026 < 0.05$, so reject $H_0$.

There is convincing evidence that the new checkout page has a different (higher) conversion rate than the old one; because visitors were randomly assigned, the page itself caused the improvement.

Worked example

Estimate the size of the improvement with a 95% confidence interval for $p_B – p_A$.

Step 1 Unpooled standard error: $\sqrt{\dfrac{0.065 \times 0.935}{2400} + \dfrac{0.05 \times 0.95}{2400}} = \sqrt{0.00002532 + 0.00001979} = \sqrt{0.00004511} \approx 0.006717$.

Step 2 Margin of error: $1.96 \times 0.006717 \approx 0.01317$.

Step 3 Interval: $0.015 \pm 0.0132$, which gives $(0.0018,\ 0.0282)$.

Step 4 Check against the test: the interval does not contain 0, which agrees with rejecting $H_0$ at the 5% level.

We are 95% confident that the new page raises the conversion rate by between about 0.2 and 2.8 percentage points.

Worked example

A health clinic runs a stop-smoking trial. Of 200 smokers randomly assigned to a new support app, 58 had quit after six months. Of 200 assigned to standard leaflets, 40 had quit. Test at the 5% level whether the app has a different quit rate. (The figures are illustrative.)

Step 1 $\hat p_{\text{app}} = 58/200 = 0.29$, $\hat p_{\text{leaf}} = 40/200 = 0.20$, difference $0.09$.

Step 2 Pooled: $\hat p_c = 98/400 = 0.245$. Expected counts: $200 \times 0.245 = 49$ and $200 \times 0.755 = 151$ in each group, all at least 10.

Step 3 SE: $\sqrt{0.245 \times 0.755 \times (2/200)} = \sqrt{0.0018498} \approx 0.04301$.

Step 4 $z = 0.09 / 0.04301 \approx 2.09$; two-sided $p \approx 2 \times 0.0182 = 0.036$.

Step 5 $0.036 < 0.05$: reject $H_0$.

The app group’s quit rate is significantly higher; at the 1% level, however, the result would not be significant, so the evidence is real but not overwhelming.

Make two-proportion z-tests automatic with the Practice Questions book: 1,569 questions, all fully worked.Get 1,569 questions · $9

What the answer tells us

The z statistic of 2.23 says the observed 1.5-point gap sits 2.23 standard errors above zero. Figure 1 shows what that means. If the two pages truly converted at the same rate, z would follow the standard normal curve. Only about 2.6% of the area lies beyond ±2.23, shaded in orange. Our result falls just outside the ±1.96 cut-offs that mark the 5% rejection region.

Where z = 2.23 falls under the null hypothesis (standard normal)

Null distribution of z5% cut-offs ±1.96Tail areas beyond ±2.23 (p ≈ 0.026)

Figure 1. If the two checkout pages converted equally, z would follow this curve. The observed z = 2.23 lies beyond the ±1.96 cut-offs; the two shaded tails together hold about 2.6% of the area, which is the two-sided p-value.

Statistical versus practical significance. The test says the difference is unlikely to be pure chance. It does not say the difference is large. The confidence interval does that job: the true lift could be as small as 0.2 points or as large as 2.8 points. For a shop with 100,000 checkout visitors a month, that range means somewhere between about 180 and 2,800 extra sales a month. Whether the redesign is worth it depends on that range, not on the p-value alone.

Reasonableness check. A quick estimate: with rates around 6%, the standard error of one proportion based on 2,400 people is about $\sqrt{0.06 \times 0.94 / 2400} \approx 0.0048$. For a difference of two such proportions, multiply by $\sqrt 2$ to get about 0.0068. A gap of 0.015 is a little over two of those, so a z of about 2.2 makes sense.

Sensitivity to sample size. Suppose the shop had stopped the test after one week, with exactly the same conversion rates but only 1,200 visitors per page (60 and 78 purchases). The pooled proportion is still 0.0575, but the standard error grows to about 0.00950. Then $z = 0.015/0.0095 \approx 1.58$ and the two-sided p-value is about 0.11. The same 1.5-point lift would not be significant. Small differences need big samples to detect. That is why A/B tests should fix the sample size in advance and not stop as soon as the result “looks good”.

Link to the chi-square test. You can also lay the data out as a 2 × 2 table (page by purchase) and run a chi-square test. For a two-sided alternative the chi-square statistic equals $z^2$: here $2.232^2 \approx 4.98$, with the same p-value.^[4] The z version has one advantage: it can test a one-sided alternative.

Common mistakeUsing the pooled proportion in the confidence interval, or the unpooled standard error in the test. The test assumes $H_0$ is true, so it pools. The interval makes no such assumption, so it does not pool. With large samples the two standard errors are close (0.006720 and 0.006717 here), but examiners expect the correct one.

Common mistakeWriting hypotheses about the sample proportions, such as $H_0: \hat p_A = \hat p_B$. Hypotheses are always about the population parameters $p_A$ and $p_B$. The sample values are known; there is nothing to test about them. Also define $p_A$ and $p_B$ in words, in context.

Exam tipStructure every answer in four parts: State the hypotheses and define the parameters; Plan, by naming the test and checking random, independence and large-counts conditions; Do the calculation, showing $\hat p_c$, the standard error, z and the p-value; Conclude in context, linking the p-value to $\alpha$. In AP Statistics free-response questions, a correct number with no conditions or no contextual conclusion loses marks. See our hypothesis testing guide for the general framework.

If you want the full method in one place, the Statistics Made Simple textbook covers one- and two-proportion inference with worked examples and tables. For drilling the calculation until it is second nature, the Statistics Made Simple Practice Questions book has hundreds of fully worked significance-test problems, including two-sample proportion questions in exam style.

Master significance tests and confidence intervals with both Statistics Made Simple books for $16.Both books · $16

Practice questions

Try each question before opening the worked answer. Use $\alpha = 0.05$ unless told otherwise.

EasyQ1. In a survey, 30 of 150 students at School P and 45 of 150 students at School Q walk to school. Find the pooled sample proportion. Show worked answer

$\hat p_c = (30 + 45)/(150 + 150) = 75/300 = 0.25$.
This is the estimate of the common proportion if the two schools have the same true rate.

EasyQ2. For the data in Q1, check the large-counts condition for a two-proportion z-test. Show worked answer

Using $\hat p_c = 0.25$: expected successes $150 \times 0.25 = 37.5$ and expected failures $150 \times 0.75 = 112.5$ in each school.
All four values are at least 10, so the condition is met. (The observed counts 30, 120, 45 and 105 are also all at least 10.)

MediumQ3. Using the data in Q1, test whether the proportion who walk differs between the two schools. Assume the samples are random and independent. Show worked answer

$H_0: p_P = p_Q$, $H_a: p_P \ne p_Q$.
$\hat p_P = 0.20$, $\hat p_Q = 0.30$, difference $= 0.10$.
SE $= \sqrt{0.25 \times 0.75 \times (1/150 + 1/150)} = \sqrt{0.0025} = 0.05$.
$z = 0.10/0.05 = 2.00$; two-sided $p = 2 \times 0.0228 \approx 0.0455$.
$0.0455 < 0.05$: reject $H_0$. There is evidence that the proportion walking differs, although the result is close to the 5% boundary.

MediumQ4. A mobile network changed its billing letter. Before the change, 132 of a random sample of 600 customers complained; after it, 99 of a separate random sample of 600 complained. Test at the 5% level whether the complaint rate has fallen. Show worked answer

$H_0: p_{\text{after}} = p_{\text{before}}$, $H_a: p_{\text{after}} < p_{\text{before}}$.
$\hat p_{\text{before}} = 0.22$, $\hat p_{\text{after}} = 0.165$, difference $= -0.055$.
$\hat p_c = 231/1200 = 0.1925$; SE $= \sqrt{0.1925 \times 0.8075 \times (2/600)} \approx 0.02276$.
$z = -0.055/0.02276 \approx -2.42$; one-sided $p = P(Z < -2.42) \approx 0.0078$.
$0.0078 < 0.05$: reject $H_0$. There is strong evidence that the complaint rate fell. (These are observational samples, so we cannot be certain the letter caused the fall.)

ExamQ5. A farmer plants 500 seeds of variety X and 500 of variety Y, assigning seeds to plots at random. 45 X seeds and 60 Y seeds fail to germinate. (a) Construct a 95% confidence interval for $p_Y – p_X$, the difference in failure rates. (b) Does the interval give evidence of a difference? Explain. (c) Carry out the matching significance test. [8] Show worked answer

(a) $\hat p_X = 0.09$, $\hat p_Y = 0.12$, difference $0.03$. [1]
Unpooled SE $= \sqrt{0.12 \times 0.88/500 + 0.09 \times 0.91/500} = \sqrt{0.0003750} \approx 0.01936$. [1]
Interval: $0.03 \pm 1.96 \times 0.01936 = 0.03 \pm 0.0380$, so $(-0.008,\ 0.068)$. [1]
(b) The interval contains 0, so a difference of zero is plausible; there is no convincing evidence of a difference at the 5% level. [1]
(c) $H_0: p_Y = p_X$, $H_a: p_Y \ne p_X$; conditions: random assignment, counts 45, 455, 60, 440 all at least 10. [1]
$\hat p_c = 105/1000 = 0.105$; SE $= \sqrt{0.105 \times 0.895 \times 2/500} \approx 0.01939$. [1]
$z = 0.03/0.01939 \approx 1.55$; two-sided $p \approx 0.122$. [1]
$0.122 > 0.05$: do not reject $H_0$; no convincing evidence that the failure rates differ, consistent with part (b). [1]

Frequently asked questions

When should I use a one-sided alternative?

Only when the research question, decided before you see the data, is about a change in one specific direction. For example, “does the new letter reduce complaints?” If you would care about a difference in either direction, use a two-sided test. The two-sided p-value is double the one-sided one, so switching after the fact makes results look stronger than they are.

Why pool the proportions at all?

Because the p-value is calculated assuming $H_0$ is true. If $p_1 = p_2$, the two groups are really one population, so the best estimate of that single proportion uses all the data at once. This gives a more stable standard error for the test.

How large do the samples need to be?

The usual rule is that each group should have at least 10 expected successes and at least 10 expected failures. Some textbooks use 5 instead of 10. With rare outcomes, such as a 1% conversion rate, you need several thousand observations per group before the normal approximation is reliable.

What is the difference between this test and a t-test?

A t-test compares means of a measured variable, and uses the t distribution because the population standard deviation must be estimated separately. For proportions, the standard error depends only on $p$ itself, so we use the standard normal distribution. Our post on t-test vs z-test explains the distinction in more detail.

References

Illowsky, B. & Dean, S. (2023) Introductory Statistics 2e. Houston, TX: OpenStax. https://openstax.org/details/books/introductory-statistics-2e
Starnes, D. S. & Tabor, J. The Practice of Statistics. New York: W. H. Freeman.
College Board. AP Statistics Course and Exam Description. New York: College Board.
Moore, D. S., McCabe, G. P. & Craig, B. A. Introduction to the Practice of Statistics. New York: W. H. Freeman.
NIST/SEMATECH. e-Handbook of Statistical Methods. National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/

Two-Proportion z-Test: Comparing Two Percentages Step by Step, with an A/B Test

The case: did the new checkout page really sell more?

The concept: comparing two proportions with a z statistic

Why this method fits the case

Setting it up

Solving it

What the answer tells us

Practice questions

Frequently asked questions

When should I use a one-sided alternative?

Why pool the proportions at all?

How large do the samples need to be?

What is the difference between this test and a t-test?

References

Keep reading

Contact Info

Quick Links

Success!