Step 2: Probability and Statistics Review for Econometrics
- Every regression number is an estimate from one sample. Judging it needs a small statistics toolkit: expected values, variances, covariances, the normal family of distributions, sampling distributions, confidence intervals and tests.
- \( \text{E}(a + bX) = a + b\,\text{E}(X) \) and \( \text{Var}(a + bX) = b^2\,\text{Var}(X) \). Variances of sums pick up a \( 2\,\text{Cov} \) term. Correlation measures only linear association.
- The conditional expectation \( \text{E}(y \mid x) \) is the population regression function: a regression line estimates it.
- The central limit theorem makes sample means (and regression coefficients) approximately normal even when the data are skewed, which is why \( t \) tests work.
- In 1,000 workers, the 95% confidence interval for the mean wage is $19.75 to $21.22, and men’s mean log wage is 0.158 above women’s (\( t = 4.92 \)).
This step is a probability and statistics review written for econometrics. If you studied statistics recently, skim the headings and do the practice questions. If it has been a while, work through it in order: the ideas here are used in every later step. All the numbers come from the same 1,000 workers in wages.csv that Step 1 used.
Random variables and their distributions
Chapter 2 of Econometrics Made Simple covers this in full, with Python, R and Stata code for every example.Get the textbook · $15
Pick a worker at random. Before you look, their wage, schooling and union status are all random variables. The simplest kind is a Bernoulli (0/1) variable. Union membership is one: 11.9% of the 1,000 workers are members, so \( P(\text{union} = 1) = 0.119 \). Econometrics calls 0/1 variables dummy variables, and Step 9 is devoted to them.
Years of education is discrete too. Figure 1 shows its distribution. The cumulative distribution function \( F(x) = P(X \le x) \) adds the probabilities from the bottom: \( P(\text{educ} \le 12) = 0.051 + 0.094 + 0.125 + 0.195 + 0.199 = 0.664 \).
wages.csv. The bars add up to 1, and adding the first five gives P(educ ≤ 12) = 0.664. Data: Econometrics Made Simple, Chapter 2.Expected value and variance
The expected value \( \text{E}(X) \) is the probability-weighted average of the possible values: the long-run average if you drew \( X \) again and again. The variance is the expected squared distance from the mean, and the standard deviation is its square root, back in the original units.
For a Bernoulli variable, \( \text{E}(X) = p \) and \( \text{Var}(X) = p(1-p) \). That is why the mean of a dummy is a share: the mean of union is 0.119, and its variance is \( 0.119 \times 0.881 = 0.1048 \).
Four rules let you handle almost anything built from simpler pieces. For constants \( a \) and \( b \):
Adding a constant shifts a distribution without changing its spread. Multiplying by \( b \) stretches the spread by \( |b| \). Note what is not on the list: in general \( \text{E}(XY) \ne \text{E}(X)\text{E}(Y) \) and \( \text{E}[\log X] \ne \log[\text{E}(X)] \).
A firm pays each worker for 40 hours a week at their hourly wage \( W \), plus a fixed $50 travel allowance, so weekly pay is \( P = 50 + 40W \). In the wage data the mean wage is $20.484 and the standard deviation is $11.864. Find the mean and standard deviation of weekly pay, and say how unusual a $40 hourly wage is.
A $40 wage is about 1.6 standard deviations above the mean. A standardised variable \( Z = (X-\mu)/\sigma \) always has mean 0 and variance 1, by the two rules above.
In a sample \( y_1, \dots, y_n \) you replace these population quantities by the sample mean \( \bar y = \frac{1}{n}\sum y_i \) and the sample variance \( s^2 = \frac{1}{n-1}\sum (y_i – \bar y)^2 \). For the wage data, the mean wage is $20.48 with \( s = \$11.86 \); the log wage has mean 2.884 and \( s = 0.514 \). The reason for \( n-1 \) comes in the section on estimators below.
Covariance, correlation and independence
Econometrics is about how variables move together. The covariance is positive when high values of \( X \) come with high values of \( Y \), negative when they come with low values.
In the wage data, the sample covariance between education and the log wage is 0.357. Its size depends on the units: measure schooling in months and it becomes \( 12 \times 0.357 = 4.28 \), though nothing has changed. Dividing by both standard deviations removes the units. The sample variances are 3.480 (education) and 0.264 (log wage), so the correlation is \( 0.357/\sqrt{3.480 \times 0.264} = 0.372 \), the figure from Step 1.
When you add two variables, their covariance enters the variance. This rule is behind the variance of a sample mean and the standard error of a difference in means:
Independence means that knowing one variable tells you nothing about the distribution of the other. Independence implies zero covariance, but not the other way round. Let \( X \) be \(-1\), 0 or 1 with probability \( \tfrac13 \) each, and \( Y = X^2 \). Then \( Y \) is completely determined by \( X \), yet \( \text{E}(X) = 0 \) and \( \text{E}(XY) = \text{E}(X^3) = 0 \), so \( \text{Cov}(X,Y) = 0 \). A straight line through a U-shape is flat.
Conditional expectation: the idea behind regression
The average log wage of all workers is 2.884. But the average among workers with 16 years of school is surely higher than among those with 8. The mean of \( y \) among units with a given value of \( x \) is the conditional expectation.
In the wage data the sample conditional means of the log wage are 2.395 at 8 years of school, 2.871 at 12 years and 3.250 at 16 years (they are the dots in Figure 1 of Step 1). They rise roughly in a straight line, which is exactly what the simple regression model assumes:
Any \( y \) can be split into its conditional mean and a deviation \( u \). The condition \( \text{E}(u \mid x) = 0 \) says that at every value of \( x \) the unobserved factors average out to zero. It is the most important assumption in econometrics: Step 4 shows that OLS is unbiased when it holds.
One more rule, the law of iterated expectations, says the overall mean is the weighted average of the conditional means: \( \text{E}(y) = \sum_j P(x = x_j)\,\text{E}(y \mid x = x_j) \). Men earn $22.23 on average and women $18.83, and men are 48.6% of the sample, so the overall mean is \( 0.486 \times 22.23 + 0.514 \times 18.83 = 10.80 + 9.68 = \$20.48 \), exactly the sample mean.
The normal distribution and its relatives
Four continuous distributions appear again and again. Each later test in this guide compares a statistic with one of them.
| Distribution | Built from | Shape | Main use |
|---|---|---|---|
| \( N(\mu, \sigma^2) \) | — | symmetric bell | large-sample tests and intervals; probit (Step 14) |
| \( \chi^2_k \) | sum of \( k \) squared standard normals | right-skewed, never negative, mean \( k \) | tests on variances; LM tests (Steps 11–12) |
| \( t_k \) | \( Z/\sqrt{V/k} \), with \( V \sim \chi^2_k \) | bell with fatter tails | tests and intervals for one coefficient (Step 5) |
| \( F_{k_1,k_2} \) | ratio of two chi-squares over their df | right-skewed, never negative | joint tests of several coefficients (Step 7) |
Two facts make the normal convenient. Standardising any normal variable gives the standard normal \( N(0,1) \), so one table answers every question. And sums of jointly normal variables are normal. For \( N(0,1) \), 95% of the probability lies between \(-1.96\) and \(1.96\), and 5% lies above 1.645: numbers you will use in nearly every step.
The \( t \) distribution matters because you almost always divide by an estimated standard deviation, which adds uncertainty and fattens the tails. The two-sided 5% critical value of \( t \) falls from 2.571 with 5 degrees of freedom to 2.042 with 30 and 1.962 with 999, close to 1.96. The distributions are linked: \( 1.96^2 = 3.84 \) is the 5% critical value of \( \chi^2_1 \), and \( 2.042^2 = 4.17 \) is that of \( F_{1,30} \).
Wages are skewed, but the log wage is close to symmetric. Suppose log wages are \( N(2.884,\ 0.514^2) \). What share of workers would earn more than $30 an hour? Compare with the data.
The normal model predicts that 15.7% of workers earn over $30. The actual share is 16.1%, so the normal approximation to the log wage works well.
Estimators: unbiased, efficient, consistent
A population parameter, such as the mean wage \( \mu \), is a fixed but unknown number. You learn about it from a random sample. An estimator is a rule for turning the sample into a guess, such as \( \bar y \); an estimate is the number it produces in one sample, such as \( \bar y = 20.48 \). Draw another sample and the estimate changes, so an estimator is itself a random variable with a sampling distribution. We judge it by three properties.
| Property | Meaning | The sample mean |
|---|---|---|
| Unbiased | \( \text{E}(\hat\theta) = \theta \): right on average | \( \text{E}(\bar y) = \mu \) |
| Efficient | smallest variance among unbiased estimators of its type | \( \text{Var}(\bar y) = \sigma^2/n \), smallest among linear unbiased estimators |
| Consistent | gets arbitrarily close to \( \theta \) as \( n \) grows | \( \text{sd}(\bar y) = \sigma/\sqrt{n} \to 0 \) |
The variance result comes straight from the rule for sums: with independent observations all covariances are zero, so \( \text{Var}(\bar y) = \frac{1}{n^2}(n\sigma^2) = \sigma^2/n \). To halve the standard deviation of the mean you need four times as many observations.
Econometrics Made Simple checks all three properties by simulation. It treats the 1,000 workers as a population (mean log wage 2.884), draws 10,000 samples of 25, and computes three estimators in each:
| Estimator | Average of 10,000 estimates | Standard deviation |
|---|---|---|
| Sample mean | 2.887 | 0.1028 |
| Sample median | 2.872 | 0.1230 |
| First observation only | 2.888 | 0.5153 |
The mean and the first observation are both unbiased, but the mean is five times more precise; the median is slightly low because the log wage is a little skewed. The \( s^2 \) that divides by \( n-1 \) averaged 0.2643, close to the population variance of 0.2638; dividing by \( n \) gave 0.2537, too small by roughly \( 24/25 \). That is why \( s^2 \) uses \( n-1 \): deviations are measured from \( \bar y \), which is fitted to the data and so sits closer to the observations than the true \( \mu \) does.
Practise until it sticks: Econometrics Made Simple Practice Questions has over 1,150 questions, each with a worked answer.Get the practice book · $9
The law of large numbers and the central limit theorem
“Any population” is the remarkable part. Hourly wages are far from normal: bounded below, bunched at low values, with a long right tail (skewness 2.40). Yet the average of a sample of wages is approximately normal once the sample is moderately large. Figure 2 re-runs the book’s simulation: 10,000 samples drawn from the 1,000 wages.
| Sample size | Mean of the sample means | sd of the sample means | \( \sigma/\sqrt{n} \) | Skewness |
|---|---|---|---|---|
| 2 | 20.505 | 8.305 | 8.385 | 1.50 |
| 10 | 20.521 | 3.762 | 3.750 | 0.69 |
| 50 | 20.490 | 1.708 | 1.677 | 0.38 |
Every row is centred on the population mean of 20.48 (the law of large numbers), the spread matches \( \sigma/\sqrt{n} \), and the skewness falls towards zero (the CLT). Regression coefficients are weighted averages of the data, so they too are approximately normal in large samples. How large is “large” depends on the skewness: for a symmetric variable 30 observations is plenty; for something as skewed as wealth you may need several hundred.
Confidence intervals
A point estimate such as \( \bar y = 20.48 \) will almost never equal \( \mu \) exactly. A confidence interval gives a range of plausible values. By the CLT, \( \bar y \) is approximately \( N(\mu, \sigma^2/n) \). Replacing \( \sigma \) with \( s \) gives the standard error \( \text{se}(\bar y) = s/\sqrt{n} \), and the standardised mean then follows a \( t_{n-1} \) distribution.
Find 95% confidence intervals for the population mean hourly wage and for the mean log wage, using \( n = 1{,}000 \).
What does “95% confidence” mean? The population mean is fixed; the interval is random. If you drew many samples and built an interval from each, about 95% of them would contain \( \mu \). The practical reading is “we are 95% confident that the mean wage lies between $19.75 and $21.22”.
Hypothesis testing
A test asks whether the data are consistent with a claim about a parameter, the null hypothesis \( H_0 \). It works like a court case: \( H_0 \) is presumed true and is rejected only if the data would be very unlikely under it. Suppose a national survey reports a mean log wage of 2.85. Test \( H_0: \mu = 2.85 \) against \( H_1: \mu \ne 2.85 \).
At the 5% level the two-sided critical value of \( t_{999} \) is 1.962. Since \( 2.090 > 1.962 \), reject \( H_0 \). At 1% the critical value is 2.581, so we do not reject there. The \( p \)-value is the probability, if \( H_0 \) were true, of a statistic at least this extreme: \( p = 0.0369 \). Reject at level \( \alpha \) whenever \( p < \alpha \). Notice that the 95% interval \( [2.852, 2.916] \) excludes 2.85: a 95% interval holds exactly the values a 5% two-sided test would not reject.
One-sided tests. If theory says which direction matters, put the claim in the alternative and all of \( \alpha \) in one tail. An adviser claims the mean wage is above $20: test \( H_0: \mu = 20 \) against \( H_1: \mu > 20 \). Then \( t = (20.484-20)/0.3752 = 1.289 \), below the one-sided 5% critical value of 1.646, with \( p = 0.0988 \). Do not reject at 5%, although you would at 10%.
Errors and power. A Type I error rejects a true \( H_0 \) (probability \( \alpha \)). A Type II error fails to reject a false one. Power is the probability of correctly rejecting a false null. If the true mean log wage were 2.90, the test above would reject at 5% with probability about 0.87; with a quarter of the sample the standard error would double and power would fall sharply. That is why “not significant” should never be read as “no effect”.
Comparing two means: the gender wage gap
For two independent samples the variance of the difference is the sum of the variances, which gives the Welch (unequal-variances) \( t \) test:
The mean log wage is 2.9653 for the 486 men (\( s = 0.5152 \)) and 2.8071 for the 514 women (\( s = 0.5010 \)). Test for a difference at the 1% level and interpret its size.
The gap is real in the population. It is not proof of discrimination: men and women may differ in experience, occupation or region. Step 9 compares them holding those factors fixed.
The textbook and the practice book together: the Econometrics Made Simple bundle for $19 (was $24).Both books · $19
Practise: 6 questions
Try each one on paper first, then check the worked answer.
EasyQ1. \( X \) takes the values 0, 1, 2 and 3 with probabilities 0.1, 0.4, 0.3 and 0.2. Find \( \text{E}(X) \), \( \text{Var}(X) \) and \( \text{sd}(X) \). Show worked answer
\( \text{E}(X^2) = 0 + 0.4 + 4(0.3) + 9(0.2) = 3.4 \).
\( \text{Var}(X) = 3.4 – 1.6^2 = 0.84 \), and \( \text{sd}(X) = \sqrt{0.84} = 0.917 \).
EasyQ2. Monthly sales \( X \) have mean 20 and standard deviation 4 (thousand units). Revenue is \( Y = 100 + 5X \). Find \( \text{E}(Y) \), \( \text{sd}(Y) \) and \( \text{Var}(Y) \). Show worked answer
\( \text{sd}(Y) = |5| \times 4 = 20 \), so \( \text{Var}(Y) = 400 \).
The constant 100 moves the mean but not the spread.
MediumQ3. \( \text{Var}(X) = 9 \), \( \text{Var}(Y) = 16 \) and \( \text{Corr}(X,Y) = 0.25 \). Find \( \text{Cov}(X,Y) \), \( \text{Var}(X + Y) \) and \( \text{Var}(2X – Y) \). Show worked answer
\( \text{Var}(X + Y) = 9 + 16 + 2(3) = 31 \).
\( \text{Var}(2X – Y) = 4(9) + 16 + 2(2)(-1)(3) = 36 + 16 – 12 = 40 \).
MediumQ4. \( X \sim N(100, 225) \). Find (a) \( P(X > 130) \) and (b) \( P(85 < X < 115) \). Show worked answer
(a) \( P(Z > (130-100)/15) = P(Z > 2) = 0.0228 \).
(b) \( P(-1 < Z < 1) = 0.6827 \).
MediumQ5. A random sample of 81 shops has mean weekly footfall 24 (hundreds) with \( s = 9 \). (a) Build a 95% confidence interval for the population mean (use \( c = 1.990 \)). (b) Test \( H_0: \mu = 26 \) against a two-sided alternative at 5%. Show worked answer
(b) \( t = (24-26)/1 = -2.00 \). \( |t| = 2.00 > 1.990 \), so reject \( H_0 \) at 5%, only just (\( p = 0.049 \)). Consistently, 26 lies just outside the interval.
ExamQ6. In wages.csv, the 648 urban workers have mean log wage 2.9180 (\( s = 0.5104 \)) and the 352 rural workers 2.8214 (\( s = 0.5149 \)). (a) State the hypotheses for a two-sided test of equal population means. [1] (b) Compute the Welch \( t \) statistic. [3] (c) Decide at the 1% level, using a critical value of 2.58. [1] (d) Express the gap as an exact percentage and say why it need not be a causal effect of living in a city. [3] Show worked answer
(b) Difference \( = 2.9180 – 2.8214 = 0.0966 \). [1] \( \text{se} = \sqrt{0.5104^2/648 + 0.5149^2/352} = 0.0340 \). [1] \( t = 0.0966/0.0340 = 2.84 \). [1]
(c) \( 2.84 > 2.58 \): reject at 1% (\( p = 0.0046 \)). [1]
(d) \( 100(e^{0.0966}-1) = 10.1\% \) higher in cities. [1] Urban and rural workers may differ in education, occupation or experience [1], and higher city wages may partly offset higher living costs or reflect who chooses to move to cities (selection). [1]
Total: 8 marks.
Check yourself
1. If \( \text{Var}(X) = 4 \), then \( \text{Var}(3X + 2) \) is
2. The central limit theorem says that, for large \( n \),
3. A two-sided test gives \( p = 0.03 \). You should
Your score: 0 out of 3
The short version
- A random variable has a distribution: a pmf for discrete variables, a density (probability = area) for continuous ones.
- \( \text{E}(a + bX) = a + b\,\text{E}(X) \), \( \text{Var}(a + bX) = b^2\,\text{Var}(X) \), and variances of sums include \( 2\,\text{Cov} \) terms. Zero correlation does not imply independence.
- \( \text{E}(y \mid x) \) is the population regression function; the error is \( y – \text{E}(y \mid x) \), with \( \text{E}(u \mid x) = 0 \).
- The normal, \( t \), \( \chi^2 \) and \( F \) distributions supply critical values; \( t_k \) approaches \( N(0,1) \) as \( k \) grows.
- \( \bar y \) is unbiased and consistent, with \( \text{sd}(\bar y) = \sigma/\sqrt{n} \). By the CLT it is approximately normal even for skewed data such as wages.
- 95% interval: \( \bar y \pm c \cdot \text{se} \). Test: \( t = (\text{estimate} – \text{hypothesised value})/\text{se} \); reject when \( p < \alpha \).
- Chapter 2 of Econometrics Made Simple has the full code, more worked examples and 21 exercises with answers.
Need a fuller statistics course first? The Statistics Made Simple bundle covers all of this in depth. Studying more than one subject? The Complete Book Bundle includes every Economics, Statistics, Mathematics and Econometrics book.
Further reading on EconTutorials
Related in-depth articles from our blog: