Econometrics guide, Unit 4, step 15 of 20Previous step Next step

Step 15: Panel Data: Fixed and Random Effects

By the EconTutorials team. 14-minute read with 6 practice questions.

SyllabusUndergraduate econometrics · Wooldridge, Introductory Econometrics, Ch. 13–14

In 30 seconds

A panel follows the same $ N $ units (people, firms, countries) over $ T $ periods. That lets you compare each unit with itself.
In $ y_{it} = \beta_1x_{it} + a_i + u_{it} $, the unobserved effect $ a_i $ holds everything fixed about unit $ i $. If it is correlated with $ x $, pooled OLS is biased.
First differences and fixed effects (the within estimator) both remove $ a_i $. Fixed effects equals OLS with a dummy for every unit. Add year effects for common shocks.
Random effects assumes $ a_i $ is uncorrelated with every $ x $. The Hausman test compares FE and RE; a rejection favours FE.
Exam point: cluster standard errors by unit, and remember what FE cannot do: time-invariant regressors, slowly changing variables, measurement error, reverse causality.

Panel data and why they help

Countries that invest more grow faster. But they also differ in institutions, geography, legal systems and culture, most of which are hard to measure and change only slowly. In a single cross-section all of these sit in the error, and if any is related to investment, the regression of growth on investment is biased. A panel offers a way out: did growth rise in the years when this country invested more than its own average? That comparison removes every fixed characteristic of the country, measured or not.

Key termA panel (longitudinal) dataset follows the same $ N $ units over $ T $ periods; observation $ (i,t) $ is unit $ i $ in period $ t $. A balanced panel has all $ NT $ observations; an unbalanced one has gaps. Software needs long format: one row per unit–period, with a unit identifier and a period identifier.

Chapter 15 of Econometrics Made Simple covers this in full: first differences, the within estimator, two-way fixed effects, random effects, the Hausman test and clustering.Get the textbook · $15

The running example is the book’s countries_panel.csv: 60 countries over 2000–2019, so 1,200 rows. It is simulated from

\[ \text{growth}_{it} = \dots + 0.11\,\text{investment}_{it} + 0.25\,(\text{schooling}_{it} – 8) + 0.012\,\text{openness}_{it} – 0.06\,\text{inflation}_{it} + a_i + (\text{2009 shock}) + u_{it}, \]

where the country effect $ a_i $ is larger in countries with higher investment and schooling, and every country’s growth was cut by 3.5 points in 2009. The true coefficients are known, so we can see which estimators get close.

Common mistakeTreating 1,200 rows as 1,200 independent observations. Twenty years of the same country share its institutions, geography and history. The two consequences are biased coefficients and standard errors that are too small.

The unobserved-effects model and pooled OLS

Write the model as

\[ y_{it} = \beta_0 + \beta_1 x_{it} + a_i + u_{it}, \]

with $ a_i $ the unobserved effect (unit effect, fixed effect) and $ u_{it} $ the idiosyncratic error that changes over time. Pooled OLS stacks all $ NT $ rows and ignores the structure, so its error is $ v_{it} = a_i + u_{it} $. If $ \text{Cov}(x_{it}, a_i) \neq 0 $, this is omitted variable bias with $ a_i $ as the omitted variable:

\[ \operatorname{plim}\hat\beta_1^{\,\text{pooled}} = \beta_1 + \frac{\text{Cov}(x_{it}, a_i)}{\text{Var}(x_{it})}. \]

Pooled OLS gives an investment coefficient of 0.140 (95% interval 0.118 to 0.163), against a true 0.11, and a schooling coefficient of 0.433 against a true 0.25. Both biases are upward, because both variables are positively related to the country effect. A between regression of average growth on average investment across the 60 countries gives 0.233: it uses only differences between countries and is even further off.

Worked example

Country A has $ x = 1, 2, 3 $ and $ y = 5, 6, 8 $; country B has $ x = 4, 5, 6 $ and $ y = 10, 12, 13 $. Find the pooled slope and the within slope.

Step 1Pooled: $ \bar{x} = 3.5 $, $ \bar{y} = 9 $. $ \sum(x-\bar{x})(y-\bar{y}) = 30 $ and $ \sum(x-\bar{x})^2 = 17.5 $, so the slope is $ 30/17.5 = 1.714 $.

Step 2Within: subtract each country’s means (A: 2 and 6.333; B: 5 and 11.667). Demeaned $ x $ is $ -1, 0, 1 $ in both. Demeaned $ y $ is $ -1.333, -0.333, 1.667 $ for A and $ -1.667, 0.333, 1.333 $ for B.

Step 3Within slope: $ (1.333 + 1.667 + 1.667 + 1.333)/(1+1+1+1) = 6/4 = 1.5 $.

Step 4Country effects: $ \hat{a}_A = 6.333 – 1.5 \times 2 = 3.333 $; $ \hat{a}_B = 11.667 – 1.5 \times 5 = 4.167 $.

Country B has both higher $ x $ and a higher effect. Pooled OLS credits part of B’s higher effect to its higher $ x $, so the pooled slope is larger.

First differences

With two periods, write the model for each and subtract. The unit effect appears in both and cancels:

\[ \Delta y_i = \delta_0 + \beta_1\,\Delta x_i + \Delta u_i . \]

OLS on this is the first-difference (FD) estimator, and $ a_i $ may be correlated with $ x $ in any way at all. With more periods, difference adjacent years and pool the $ N(T-1) $ changes, usually with year dummies. The price: FD needs strict exogeneity ($ u_{it} $ unrelated to $ x $ in every period), and it throws away all between variation. A variable that never changes drops out; one that changes slowly is estimated imprecisely.

In the country panel, using only 2018 and 2019 gives 60 differences, an investment estimate of 0.092 and an absurd schooling estimate of 3.90: schooling rose by only 0.05 years on average between the two years, so there was almost nothing to learn from. All 20 years give 1,140 differences and an investment estimate of 0.073 (clustered standard error 0.029), with 0.11 inside the interval.

Fixed effects: the within estimator

The more common way to remove $ a_i $ is to subtract each unit’s average over time. Averaging leaves $ a_i $ unchanged, so subtracting removes it:

\[ \ddot{y}_{it} = \beta_1\,\ddot{x}_{it} + \ddot{u}_{it}, \qquad \ddot{y}_{it} = y_{it} – \bar{y}_i, \quad \ddot{x}_{it} = x_{it} – \bar{x}_i . \]

OLS on the demeaned data is the fixed-effects (FE) or within estimator. It is unbiased under strict exogeneity and time variation in each regressor. An equivalent route gives each unit its own intercept: regress $ y $ on $ x $ and 59 country dummies, the least squares dummy variable (LSDV) regression. The slopes are identical to every digit.

Key termThe fixed-effects estimator is OLS on data demeaned by unit, or equivalently OLS with a dummy for every unit (LSDV). Its residual degrees of freedom are $ NT – N – k $, because one mean is estimated per unit. Demeaning by hand and running ordinary OLS gets the degrees of freedom wrong, so its standard errors are slightly too small; use a panel command.

In the country panel FE gives an investment coefficient of 0.0875 (standard error 0.023), much closer to 0.11. Read it as a within-country effect: in years when a country’s investment share is one point above its own average, its growth is about 0.09 points above its own average, holding the other regressors fixed. It says nothing about why some countries invest more than others on average; that comparison is exactly what FE throws away. The estimated country effects have a correlation of 0.58 with average investment, the correlation that biased pooled OLS. In Python, PanelOLS from linearmodels with entity_effects=True; in R, plm(..., model = "within"); in Stata, xtreg y x, fe after xtset.

Exam tipWhen asked why FE differs from pooled OLS, give one chain: (1) name a time-invariant unobserved factor; (2) say how it affects $ y $; (3) say how it is related to $ x $; (4) conclude the direction of the pooled bias; (5) note that FE removes it because it does not change over time. With $ T = 2 $, FE and FD give identical slopes; with more periods, report both, and suspect a failure of strict exogeneity if they differ a lot.

Two-way fixed effects: adding year effects

Country effects remove factors that differ across countries but not over time. Many shocks do the reverse: a global recession or an oil-price jump hits everyone in the same year. If such a shock is correlated with the regressors, leaving it out biases the slope. Add one intercept per year:

\[ y_{it} = \beta_1 x_{it} + a_i + \lambda_t + u_{it}. \]

This is the two-way fixed-effects model, the workhorse of applied panel work. Difference-in-differences (Step 16; see also Difference-in-Differences Explained) is a special case, with $ x_{it} $ a dummy for being treated after a policy change.

Year effects from two-way fixed effects, relative to 2000 60 countries, 2000–2019

estimated year effect2009 global shock

Figure 1. The year dummies absorb shocks common to every country. The 2009 effect is −3.82 points, close to the −3.5 built into the simulation; the other years lie between about −0.9 and −0.15. Data: countries_panel.csv, recomputed.

The investment coefficient moves from 0.088 to 0.078 (clustered standard error 0.022). The schooling coefficient tells a different story: its clustered standard error almost quadruples, from 0.10 to 0.38. Within each country schooling rises almost in a straight line, and that common upward drift is now absorbed by the year effects. What remains is the small wobble around each country’s trend: the slowly changing variable problem in its sharpest form.

Random effects and the Hausman test

FE allows any correlation between $ a_i $ and the regressors, but discards all between-unit variation and every time-invariant regressor. Random effects (RE) assumes more: $ a_i $ is uncorrelated with the regressors in every period. Then pooled OLS is consistent but inefficient, because the composite error $ a_i + u_{it} $ is correlated across a unit’s periods with correlation $ \sigma_a^2/(\sigma_a^2 + \sigma_u^2) $. RE is the GLS fix: subtract a fraction $ \theta $ of each unit’s mean,

\[ y_{it} – \theta\bar{y}_i = \beta_0(1-\theta) + \beta_1(x_{it} – \theta\bar{x}_i) + (v_{it} – \theta\bar{v}_i), \qquad \theta = 1 – \sqrt{\frac{\sigma_u^2}{\sigma_u^2 + T\sigma_a^2}} . \]

If $ \theta = 0 $, RE is pooled OLS; as $ \theta \to 1 $ it becomes FE. RE can estimate time-invariant regressors such as region, because it does not remove the whole mean.

Worked example

An RE model of the country panel estimates $ \hat\sigma_u^2 = 3.083 $ and $ \hat\sigma_a^2 = 0.078 $, with $ T = 20 $. Compute $ \theta $ and say what it means.

Step 1$ \theta = 1 – \sqrt{3.083/(3.083 + 20 \times 0.078)} = 1 – \sqrt{3.083/4.643} = 1 – 0.815 = 0.185 $. (Software, with unrounded variances, prints 0.186.)

Step 2RE subtracts less than a fifth of each country mean, so it is close to pooled OLS and keeps most of the between variation.

A small $ \hat\sigma_a^2 $ is not evidence that country effects are unimportant. Here the biased coefficients on investment and schooling have soaked up much of the between-country variation, which is exactly the correlation RE rules out.

If the RE assumption holds, FE and RE are both consistent and RE is more efficient. If it fails, only FE is consistent. So a large gap between them is evidence against RE. That is the Hausman test (Hausman, 1978). For one coefficient,

\[ H = \frac{(\hat\beta^{\,\text{FE}} – \hat\beta^{\,\text{RE}})^2}{\text{se}(\hat\beta^{\,\text{FE}})^2 – \text{se}(\hat\beta^{\,\text{RE}})^2} \sim \chi^2_1 \ \text{under } H_0 , \]

and with $ k $ coefficients the same idea gives a $ \chi^2_k $ statistic. Both models must use conventional (not clustered) standard errors for this form. In the country panel, with year effects, RE gives an investment coefficient of 0.131, close to pooled OLS and 0.053 above FE. The four-coefficient Hausman statistic is $ H = 10.66 $ against the $ \chi^2_4 $ critical value 9.49 ($ p = 0.031 $): reject, so FE is preferred. Our article Fixed Effects vs Random Effects walks through the choice in more detail.

Common mistakeReading a Hausman test that fails to reject as proof that RE is right. It may only mean the FE estimates are too imprecise to show a difference. In these data the same test without year effects gives $ H = 8.09 $, $ p = 0.088 $: the unmodelled 2009 shock adds noise that hides the gap. For causal questions FE is the safe default.

Clustered standard errors

FE removes $ a_i $, but a country’s idiosyncratic shocks may still persist over several years. If the regressor is persistent too, ordinary standard errors, which treat every row as fresh information, are too small. Cluster-robust standard errors allow any correlation within each unit and any heteroskedasticity, assuming only independence between units. For the within estimator with one regressor,

\[ \text{se}_{\text{cl}}(\hat\beta_1) = \frac{\sqrt{\sum_{i=1}^{N}\big(\sum_{t=1}^{T}\ddot{x}_{it}\,\hat{u}_{it}\big)^2}}{\sum_{i}\sum_{t}\ddot{x}_{it}^{\,2}} , \]

usually with a small-sample factor. It is the robust formula of Step 11 with the products summed within each country before squaring. In the book’s simulation with persistent $ x $ and $ u $, the true standard deviation of $ \hat\beta_1 $ is 0.053; the usual standard error averages 0.030, little more than half, while the clustered one averages 0.054. Clustering needs many clusters: 60 countries is fine; fewer than about 30 is unreliable.

Five estimates of the effect of investment on growth 95% intervals, clustered by country

uses between-country variationremoves the country effectstrue value

Figure 2. Pooled OLS and random effects lean on comparisons between countries and are biased upwards; both intervals exclude 0.11. The three estimators that remove the country effects are less precise but centred near the truth. Numbers from Econometrics Made Simple, Chapter 15.

Figure 2 collects the investment estimates, all with clustered standard errors. Pooled OLS and RE, which rely on between-country variation, are biased upward and look precise. FD, FE and two-way FE are less precise but centred near the truth: fixed effects buys consistency with precision.

What fixed effects cannot do

Time-invariant regressors are wiped out. A dummy for Europe is perfectly collinear with the country effects; software refuses to estimate it.
Slowly changing variables are imprecise. Schooling has a total standard deviation of 2.37 years but a within-country one of only 0.48, so FE keeps about a fifth of its spread. Investment keeps about half (2.40 of 5.04).
Measurement error is magnified. Demeaning removes much of the true variation but none of the noise. Adding noise with a standard deviation of 2.5 to investment cuts the pooled estimate by about 22% (0.140 to 0.110) but more than halves FE (0.088 to 0.041).
Reverse causality and time-varying omitted factors remain. If a growth boom raises next year’s investment, strict exogeneity fails. That needs instruments (Step 13) or a policy experiment.

A practical sequence: describe $ N $, $ T $, balance and how much of each variable varies within units; estimate pooled OLS as a benchmark, then two-way FE, always clustering by unit; if they differ a lot, FE is the credible one, and a Hausman test formalises it; use RE only with a good reason to believe its assumption; and discuss honestly what FE cannot handle.

Practise choosing between pooled OLS, FE and RE with the Practice Questions book, with worked answers to every question.1,150+ questions · $9

Practise: 6 questions

Try each one on paper first, then check the worked answer.

EasyQ1. Three firms are observed in two years. Training hours change by $ \Delta x = 2, 1, 3 $ and output by $ \Delta y = 4, 1, 5 $. Estimate the first-difference regression $ \Delta y = \delta_0 + \beta_1\Delta x $. Show worked answer

Means: $ \overline{\Delta x} = 2 $, $ \overline{\Delta y} = 3.333 $.
Deviations of $ \Delta x $: $ 0, -1, 1 $; of $ \Delta y $: $ 0.667, -2.333, 1.667 $.
$ \hat\beta_1 = (0 + 2.333 + 1.667)/(0 + 1 + 1) = 4/2 = 2.0 $.
$ \hat\delta_0 = 3.333 – 2 \times 2 = -0.667 $: the common change in output between the years, holding training fixed.

EasyQ2. In a panel with $ T = 8 $, $ \hat\sigma_u^2 = 2 $ and $ \hat\sigma_a^2 = 1.5 $. Compute the random-effects $ \theta $. Is RE closer to pooled OLS or to FE? Show worked answer

$ \theta = 1 – \sqrt{2/(2 + 8 \times 1.5)} = 1 – \sqrt{2/14} = 1 – 0.378 = 0.622 $.
RE subtracts about 62% of each unit mean: it sits nearer to FE than to pooled OLS, because the unit effects are large relative to the idiosyncratic noise.

MediumQ3. For the effect of class size on test scores, FE gives $ -0.45 $ (se 0.10) and RE gives $ -0.62 $ (se 0.06). Carry out the Hausman test at 5% and state which estimator you prefer. Show worked answer

$ H = (-0.45 + 0.62)^2/(0.10^2 – 0.06^2) = 0.0289/0.0064 = 4.52 $.
The 5% critical value of $ \chi^2_1 $ is 3.84; $ 4.52 > 3.84 $, so reject ($ p = 0.034 $).
The school effects appear correlated with class size, so RE is inconsistent: prefer FE.

MediumQ4. A student demeans a panel of $ N = 100 $ firms over $ T = 5 $ years by hand and runs ordinary OLS with $ k = 3 $ regressors, getting a standard error of 0.040. What is the correct FE standard error? Show worked answer

Hand OLS uses $ NT – k = 500 – 3 = 497 $ degrees of freedom; the correct number is $ NT – N – k = 397 $.
Correct se $ = 0.040 \times \sqrt{497/397} = 0.040/0.894 = 0.0448 $.
The hand-made standard error is about 11% too small. (Clustering by firm would be better still.)

MediumQ5. In a two-way FE wage regression on a panel of workers (worker and year effects), which of these can be estimated: (a) gender; (b) union membership; (c) years of potential experience, which rises by exactly one each year for everyone; (d) region of birth? Explain. Show worked answer

(a) No: constant within each worker, absorbed by the worker effect.
(b) Yes, from workers who join or leave a union (though measurement error in union status is magnified).
(c) No: within a worker it rises one-for-one with the year, so once worker and year effects are included it is perfectly collinear with them.
(d) No: time-invariant, like gender. RE, or adding worker means of the time-varying regressors, would be needed.

ExamQ6. Using 300 schools over 6 years, pooled OLS finds a large positive effect of spending per pupil on the pass rate; school fixed effects give a small, insignificant effect. (a) Give a reason why pooled OLS could be biased, and its direction. [3] (b) Give a different reason why the FE estimate could be small and imprecise. [2] (c) How should the standard errors be computed, and why? [2] Show worked answer

(a) Schools with richer catchment areas (a time-invariant factor) [B1] both spend more and get better results [B1], so $ \text{Cov}(x, a_i) > 0 $ and pooled OLS is biased upwards; FE removes this. [B1]
(b) Spending per pupil changes little within a school from year to year, so FE has little variation to work with and the estimate is imprecise. [B1] Any measurement error in spending is also magnified, biasing FE towards zero. [B1]
(c) Clustered by school [B1], because a school’s errors are likely to be correlated across years, which makes conventional standard errors too small. [B1]
Total: 7 marks.

Check yourself

1. The fixed-effects estimator removes

Demeaning by unit removes anything fixed over time. Year effects handle common shocks; time-varying omitted factors remain.

2. A Hausman test rejects its null hypothesis. You should prefer

$ H_0 $ is that RE is consistent. Rejection means the unit effects are correlated with the regressors, so only FE is consistent.

3. In a panel of countries, standard errors should usually be clustered by

A country’s errors are likely to be correlated across its years; clustering by country allows for that.

Your score: 0 out of 3

The short version

A panel follows the same units over time; use long format with unit and period identifiers.
Pooled OLS is biased when the unit effect $ a_i $ is correlated with $ x $: 0.140 against a true 0.11 for investment.
First differences and fixed effects remove $ a_i $. FE equals LSDV; its coefficients are within-unit effects. Two-way FE adds year effects (2009: −3.82).
Random effects quasi-demeans by $ \theta $ and needs $ a_i $ uncorrelated with $ x $. Hausman: $ H = 10.66 $, $ p = 0.031 $, so FE.
Cluster by unit. FE cannot estimate time-invariant variables, is imprecise for slow ones, magnifies measurement error and does not fix reverse causality.
The full chapter in Econometrics Made Simple has the code in Python, R and Stata, the clustering simulation and a complete set of exercises with answers.

Textbook and practice questions together: get both Econometrics Made Simple books for $19 (was $24).Both books · $19

Studying more than one subject? The Complete Book Bundle gives you every Economics, Statistics, Mathematics and Econometrics book in one download.