Ordinary Least Squares (OLS): Derivation, Assumptions, and Why Omitted Variable Bias Ruins Everything

Introduction: The Workhorse of Empirical Economics

Almost every empirical claim you have ever read in economics — that education raises earnings, that minimum wages do or do not cost jobs, that carbon taxes reduce emissions — rests on a regression. Usually an ordinary least squares regression.

OLS is simple enough to derive in a page and subtle enough that misunderstanding it produces most of the bad empirical work in the social sciences. The formula is not the hard part. The hard part is knowing when the number it produces means what you want it to mean.

This guide derives the estimator, states the assumptions precisely, explains what breaks when each one fails, and connects the whole apparatus to the credibility revolution that reshaped applied economics after 1990.

1. The Model and the Estimator

The simple linear regression model posits that a dependent variable Y is generated by:

Y_i = β₀ + β₁X_i + u_i

where u_i is an unobserved error term capturing everything other than X that affects Y.

We observe (X_i, Y_i) pairs but not β₀, β₁ or u_i. We want estimates.

The minimisation problem

OLS chooses the estimates that minimise the sum of squared residuals:

min_b₀,b₁ Σᵢ (Y_i − b₀ − b₁X_i)²

Why squared, rather than absolute, deviations? Three reasons. Squaring penalises large errors disproportionately. It makes the objective function differentiable everywhere, so calculus works. And under normality it coincides with maximum likelihood.

Deriving the estimator

Take the partial derivative with respect to b₀ and set it to zero:

−2Σ(Y_i − b₀ − b₁X_i) = 0

Take the partial derivative with respect to b₁:

−2ΣX_i(Y_i − b₀ − b₁X_i) = 0

These are the normal equations. Solving them yields:

β̂₁ = Σ(X_i − X̄)(Y_i − Ȳ) / Σ(X_i − X̄)² = Cov(X,Y) / Var(X)

β̂₀ = Ȳ − β̂₁X̄

The slope estimate is the sample covariance of X and Y divided by the sample variance of X. This is worth internalising: OLS is a covariance-to-variance ratio. It measures how much X and Y move together, scaled by how much X moves on its own.

Two properties follow immediately from the normal equations:

• The residuals sum to zero: Σû_i = 0.
• The residuals are orthogonal to the regressor: ΣX_iû_i = 0.
• The regression line passes through the point of means (X̄, Ȳ).

These are mechanical consequences of the algebra. They hold in every dataset, whether or not the model is true. This is the first thing students get wrong: the fact that your residuals average zero tells you nothing about whether your model is correct.

2. The Gauss-Markov Assumptions

OLS produces numbers regardless of assumptions. Whether those numbers are good estimates depends on the following.

A1. Linearity in parameters

The model is linear in β, not necessarily in X. Y = β₀ + β₁log(X) + u is a linear model. Y = β₀ + X^β₁ + u is not.

A2. Random sampling

Observations are drawn independently from the population. Violated by time series (serial dependence) and clustered data (students within schools).

A3. No perfect collinearity

In multiple regression, no regressor is an exact linear combination of the others. Violated by the dummy variable trap: including dummies for every category and an intercept.

A4. Zero conditional mean — E(u | X) = 0

This is the assumption that matters. It states that the unobserved determinants of Y are, on average, unrelated to X at every value of X.

If it holds, β̂₁ is unbiased and consistent, and it estimates a causal effect.

If it fails, β̂₁ is biased and inconsistent, and estimates nothing in particular. No sample size fixes it. No standard error correction fixes it. Getting more data makes you more precisely wrong.

A5. Homoskedasticity — Var(u | X) = σ²

Error variance is constant across values of X. If violated (heteroskedasticity), β̂ remains unbiased but standard errors are wrong, so t-statistics and confidence intervals are invalid.

A6. Normality of errors (optional)

Needed only for exact finite-sample inference. With large samples, the Central Limit Theorem delivers asymptotic normality of β̂ regardless.

The Gauss-Markov Theorem

Under A1–A5, OLS is BLUE: the Best Linear Unbiased Estimator. “Best” means minimum variance within the class of linear unbiased estimators.

Read the qualifiers carefully. It is best among linear estimators. It is best among unbiased estimators. A biased nonlinear estimator (ridge regression, LASSO) may have lower mean squared error. Gauss-Markov does not say OLS is the best estimator; it says it wins a restricted competition.

3. Omitted Variable Bias: The Central Problem

Suppose the true model is:

Y = β₀ + β₁X + β₂Z + u

but you estimate:

Y = α₀ + α₁X + v

Then Z has been absorbed into v. If Z is correlated with X, assumption A4 fails. The bias is:

E(α̂₁) = β₁ + β₂ × δ

where δ is the coefficient from regressing Z on X.

The sign of the bias is therefore the sign of β₂ × δ:

	Corr(X, Z) > 0	Corr(X, Z) < 0
β₂ > 0	Upward bias	Downward bias
β₂ < 0	Downward bias	Upward bias

The canonical example: returns to schooling

Regress log wages on years of education. The coefficient is reliably around 0.08–0.10 — each year of schooling associates with 8–10% higher earnings.

Is that causal? Only if nothing correlated with schooling independently raises earnings. But ability plausibly does both: able people get more education and earn more conditional on education.

Here β₂ (effect of ability on wages) > 0 and δ (correlation of ability with schooling) > 0. The bias is upward. The naive OLS estimate overstates the causal return to education.

This is ability bias, and resolving it occupied labour economics for thirty years. Card’s (1999) Handbook of Labor Economics chapter surveys the resulting literature. Curiously, instrumental variable estimates that purge ability bias frequently come out higher than OLS, not lower — a puzzle usually attributed to measurement error in schooling (which biases OLS toward zero) and to local average treatment effects, since IV estimates the return for the specific subpopulation whose schooling the instrument moved.

4. What Breaks, and How to Fix It

Heteroskedasticity

Symptom: residual spread widens or narrows with X. Classic in cross-sectional data — spending variance rises with income.

Consequence: coefficients fine, standard errors wrong.

Detection: Breusch–Pagan test; White test. Or simply plot residuals against fitted values.

Fix: Use heteroskedasticity-robust (White/Huber) standard errors. Modern practice is to report robust standard errors by default and never test. The cost is negligible; the risk of not doing so is not.

Autocorrelation

Symptom: in time series, errors correlate across periods. Cov(u_t, u_t−1) ≠ 0.

Consequence: standard errors typically understated; you find significance that is not there.

Detection: Durbin–Watson statistic (values near 2 suggest no first-order autocorrelation); Breusch–Godfrey test for higher orders.

Fix: Newey–West HAC standard errors, or explicit modelling of the error process.

Clustering

Symptom: observations grouped (students in classrooms, workers in firms, individuals in states). Errors correlate within group.

Consequence: Moulton (1990) showed that ignoring clustering can understate standard errors by a large factor when the regressor varies only at the group level. Many published policy findings from the 1990s do not survive clustering.

Fix: Cluster-robust standard errors at the level of treatment assignment. Bertrand, Duflo and Mullainathan (2004) demonstrated this vividly for difference-in-differences designs, showing that serially correlated outcomes plus a serially correlated treatment produce wildly overstated significance unless clustering is addressed.

Multicollinearity

Symptom: regressors highly correlated. Coefficients unstable, standard errors large, but the model fits well overall.

Consequence: OLS remains BLUE. Nothing is biased. You simply cannot separate the individual contributions.

Detection: Variance Inflation Factor; VIF above 10 is conventionally flagged.

Fix: Often none is needed. Multicollinearity is a data problem, not a model problem. If you care about the joint effect, report it. Do not drop variables to reduce VIF — that reintroduces omitted variable bias, trading a variance problem for a bias problem.

Measurement error

In Y: Increases error variance, inflates standard errors. Coefficients unbiased.

In X: Causes attenuation bias — the coefficient is biased toward zero. Classical measurement error in a regressor makes you understate the true effect. This is not fixed by more data.

5. Interpreting Coefficients Correctly

Specification	Interpretation of β₁
Y = β₀ + β₁X	A one-unit rise in X associates with a β₁-unit rise in Y
log(Y) = β₀ + β₁X	A one-unit rise in X associates with a 100×β₁ percent rise in Y (for small β₁)
Y = β₀ + β₁log(X)	A 1% rise in X associates with a β₁/100 unit rise in Y
log(Y) = β₀ + β₁log(X)	β₁ is the elasticity of Y with respect to X

The log-log elasticity interpretation is why demand estimation almost always uses logs.

The word “associates”

Note that every interpretation above says associates, not causes. The regression coefficient becomes a causal parameter only under A4, and A4 is an assumption about unobservables, which is by construction untestable. No diagnostic test in any software package tests for omitted variable bias, because the omitted variable is not in your data.

R² and what it does not tell you

R² = 1 − SSR/SST is the fraction of variation in Y explained by the model. It is not a measure of whether your model is correct, whether your coefficients are causal, or whether your specification is appropriate.

A regression of wages on a coin flip will have R² near zero and may still be an unbiased estimate of the (null) causal effect of coin flips. A regression of ice cream sales on drowning deaths will have a high R² and identify nothing. Adding any variable weakly increases R², which is why adjusted R² exists and why neither should be used for model selection in causal work.

6. From Correlation to Causation: The Credibility Revolution

Because A4 fails routinely in observational data, applied economics reorganised itself around research designs that make A4 credible by construction.

Angrist and Pischke (2010), in the Journal of Economic Perspectives, called this the credibility revolution. The 2021 Nobel Prize to Card, Angrist and Imbens recognised it.

The core designs:

• Randomised controlled trials. Random assignment guarantees E(u|X) = 0 by design. Banerjee, Duflo and Kremer’s 2019 Nobel recognised the application of this to development economics.
• Instrumental variables. Find a variable Z that affects X but has no direct effect on Y (the exclusion restriction). Angrist and Krueger (1991) famously used quarter of birth as an instrument for schooling, exploiting compulsory schooling laws.
• Difference-in-differences. Compare treated and untreated groups before and after a policy. Requires parallel trends — the assumption that, absent treatment, the two groups would have moved together.
• Regression discontinuity. Exploit a sharp cutoff in treatment assignment. Units just above and just below the threshold are comparable.
• Fixed effects. With panel data, control for all time-invariant unobserved heterogeneity by within-transforming the data.

Each design converts a general identification problem into a specific, arguable assumption. That is the whole contribution. You cannot escape needing an assumption; you can make the assumption transparent and debatable rather than hidden.

7. A Practical Checklist

1. Before running anything, ask: what is the source of variation in X? If you cannot answer, you cannot interpret the coefficient causally.
2. Plot your data. Regression is a summary; look at what is being summarised.
3. Use robust standard errors by default. Cluster at the level of treatment assignment.
4. Report the coefficient, its standard error, and the sample size. Never report only stars.
5. Think about omitted variables explicitly. Sign the likely bias using the β₂ × δ rule.
6. Distinguish statistical significance from economic significance. A precisely estimated coefficient of 0.0001 is significant and irrelevant.
7. Do not select variables to maximise R². Do not select specifications to achieve significance.

Summary

Ordinary least squares is a covariance-to-variance ratio. It is unbiased and efficient under a set of assumptions, of which one — the zero conditional mean of the error — carries essentially all the weight and is untestable.

Everything else in modern applied econometrics is machinery built to make that one assumption believable. Heteroskedasticity and autocorrelation affect inference; they do not threaten identification. Omitted variables threaten identification, and no amount of data or robust standard errors will save you from them.

The regression is easy. The argument for why the regression identifies anything is the entire job.

---

Exercises for Further Thought

1. OLS estimates of the return to schooling suffer from upward ability bias in theory, yet instrumental variable estimates using compulsory schooling laws frequently exceed the OLS estimates rather than falling below them. Propose at least three explanations for this pattern — considering measurement error in reported schooling, heterogeneous treatment effects, and the specific subpopulation whose behaviour a compulsory schooling instrument actually changes. Which explanation do you find most persuasive, and what evidence would distinguish them?

Suggested reading: Card, D. (2001). “Estimating the Return to Schooling: Progress on Some Persistent Econometric Problems.” Econometrica, 69(5), 1127–1160. Pay particular attention to Section 4 on the interpretation of IV estimates as local average treatment effects.

2. Bertrand, Duflo and Mullainathan (2004) showed that a large fraction of published difference-in-differences studies had understated their standard errors by ignoring serial correlation, and that placebo “treatments” assigned at random would have been found statistically significant far more often than 5% of the time. This is a purely statistical error, discoverable from the published data. Why did it persist for so long in a discipline with strong incentives for referees to find errors? What does this episode imply about the reliability of the applied literature published before 2004 — and what analogous errors might be present in current practice that we have not yet identified?

Suggested reading: Bertrand, M., Duflo, E., & Mullainathan, S. (2004). “How Much Should We Trust Differences-in-Differences Estimates?” Quarterly Journal of Economics, 119(1), 249–275.

---

References

1. Angrist, J. D., & Krueger, A. B. (1991). Does Compulsory School Attendance Affect Schooling and Earnings? Quarterly Journal of Economics, 106(4), 979–1014.
2. Angrist, J. D., & Pischke, J.-S. (2009). Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton University Press.
3. Angrist, J. D., & Pischke, J.-S. (2010). The Credibility Revolution in Empirical Economics. Journal of Economic Perspectives, 24(2), 3–30.
4. Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How Much Should We Trust Differences-in-Differences Estimates? Quarterly Journal of Economics, 119(1), 249–275.
5. Card, D. (2001). Estimating the Return to Schooling: Progress on Some Persistent Econometric Problems. Econometrica, 69(5), 1127–1160.
6. Moulton, B. R. (1990). An Illustration of a Pitfall in Estimating the Effects of Aggregate Variables on Micro Units. Review of Economics and Statistics, 72(2), 334–338.
7. White, H. (1980). A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity. Econometrica, 48(4), 817–838.
8. Wooldridge, J. M. (2019). Introductory Econometrics: A Modern Approach (7th ed.). Cengage.