Econometrics guide, Unit 1, step 3 of 20Previous step Next step

Step 3: Simple Regression and OLS

By the EconTutorials team. 14-minute read with 6 practice questions.

MatchesThe simple linear regression model · Wooldridge ch. 2 · Stock and Watson ch. 4

In 30 seconds

The simple regression model $ y = \beta_0 + \beta_1 x + u $ splits $ y $ into a part explained by $ x $ and an error $ u $ holding everything else. $ \beta_1 $ is the change in $ y $ for a one-unit change in $ x $, with $ u $ held fixed.
Ordinary least squares (OLS) picks the line that minimises the sum of squared residuals: $ \hat\beta_1 = \sum(x_i-\bar x)(y_i-\bar y)/\sum(x_i-\bar x)^2 $ and $ \hat\beta_0 = \bar y – \hat\beta_1\bar x $.
OLS residuals always sum to zero, are uncorrelated with $ x $, and the line passes through $ (\bar x, \bar y) $. These hold by construction; they do not test any assumption.
$ R^2 = \text{SSE}/\text{SST} $ is the share of the variation in $ y $ explained by $ x $. A high $ R^2 $ does not mean causality; a low one does not make the slope useless.
For 500 houses, $ \widehat{\text{price}} = 115.22 + 0.1408\,\text{sqft} $ with $ R^2 = 0.552 $: about $141 per square foot.

Video lesson coming soonA short animated lesson for this step will appear here.

The simple regression model

How much more does a house sell for if it is 100 square feet bigger? How much higher is the wage of a worker with one more year of school? Each question links one outcome to one explanatory variable and asks for a number. The simple regression model turns a cloud of data points into that number: a straight line with an intercept and a slope.

\[ y = \beta_0 + \beta_1 x + u \]

Key terms$ y $: the dependent variable (explained variable, regressand). $ x $: the independent or explanatory variable (regressor). $ u $: the error term, all factors other than $ x $ that affect $ y $. $ \beta_0 $: the intercept. $ \beta_1 $: the slope. The model is simple because it has one $ x $, and linear because $ y $ is linear in the parameters.

Chapter 3 of Econometrics Made Simple covers this in full, with Python, R and Stata code for every example.Get the textbook · $15

If the factors in $ u $ are held fixed ($ \Delta u = 0 $), the model says $ \Delta y = \beta_1 \Delta x $. So $ \beta_1 $ is the change in $ y $ when $ x $ rises by one unit, other things equal, and the same at every starting value of $ x $. The intercept $ \beta_0 $ is the value of $ y $ when $ x = 0 $ and $ u = 0 $. It is often meaningless in itself (no house has zero floor area), but leaving it out causes problems, as you will see below.

The key assumption: zero conditional mean

To learn $ \beta_1 $ from data, you need an assumption about how $ u $ and $ x $ are related. As long as the model has an intercept, $ \text{E}(u) = 0 $ is a harmless normalisation. The real content is this:

\[ \text{E}(u \mid x) = 0 \quad\Longrightarrow\quad \text{E}(y \mid x) = \beta_0 + \beta_1 x. \]

In words: whatever value $ x $ takes, the unobserved factors are on average the same. Then the average of $ y $ lies on a straight line, the population regression function from Step 2.

Is it plausible? In $ \text{wage} = \beta_0 + \beta_1\,\text{educ} + u $, suppose $ u $ is mostly ability. The assumption requires that people with 8, 12 and 16 years of school have the same average ability. More able people tend to stay in school longer, so the assumption probably fails, and the slope overstates the causal return to schooling. In a house-price model, $ u $ contains neighbourhood quality; if bigger houses are built in nicer neighbourhoods, the assumption fails there too. Data can never prove the assumption holds. It is an argument you make about the economics.

Common mistakeThinking $ \text{E}(u) = 0 $ is enough. With an intercept it is always true. The content of $ \text{E}(u \mid x) = 0 $ is that the average of $ u $ is the same at every value of $ x $. It implies that $ u $ and $ x $ are uncorrelated, but not the reverse.

Deriving the OLS estimators

Given a sample $ (x_i, y_i) $, $ i = 1, \dots, n $, any candidate line $ b_0 + b_1 x $ gives each observation a fitted value and a residual, the vertical gap between the point and the line. Making the residuals small on average is not enough, because positive and negative residuals cancel. OLS makes the squared residuals small.

Key termThe ordinary least squares (OLS) estimates $ \hat\beta_0, \hat\beta_1 $ are the values of $ b_0, b_1 $ that minimise the sum of squared residuals $ S(b_0, b_1) = \sum_{i=1}^n (y_i – b_0 – b_1 x_i)^2 $.

At the minimum both partial derivatives are zero. Differentiating and dividing by $ -2 $ gives the two first-order conditions, also called the normal equations:

\[ \sum_{i=1}^n \left(y_i – \hat\beta_0 – \hat\beta_1 x_i\right) = 0, \qquad \sum_{i=1}^n x_i\left(y_i – \hat\beta_0 – \hat\beta_1 x_i\right) = 0. \]

Step 1Divide the first condition by $ n $: $ \bar y – \hat\beta_0 – \hat\beta_1 \bar x = 0 $, so $ \hat\beta_0 = \bar y – \hat\beta_1 \bar x $.

Step 2Substitute into the second: $ \sum x_i (y_i – \bar y) = \hat\beta_1 \sum x_i (x_i – \bar x) $.

Step 3Deviations from a mean sum to zero, so you can subtract $ \bar x $ inside: $ \sum x_i(y_i – \bar y) = \sum (x_i – \bar x)(y_i – \bar y) $ and $ \sum x_i(x_i-\bar x) = \sum (x_i – \bar x)^2 $.

Step 4Divide, which needs $ \sum (x_i – \bar x)^2 > 0 $: $ x $ must vary.

\[ \hat\beta_1 = \frac{\sum (x_i – \bar x)(y_i – \bar y)}{\sum (x_i – \bar x)^2} = \frac{s_{xy}}{s_x^2} = r_{xy}\frac{s_y}{s_x}, \qquad \hat\beta_0 = \bar y – \hat\beta_1 \bar x. \]

The slope is the sample covariance of $ x $ and $ y $ divided by the sample variance of $ x $. The correlation form shows that the slope has the same sign as the correlation, rescaled into units of $ y $ per unit of $ x $. The same two equations also come from the population conditions $ \text{E}(u) = 0 $ and $ \text{E}(xu) = 0 $ with sample averages in place of population ones: OLS makes the residuals behave in the sample the way the errors are assumed to behave in the population.

OLS by hand: six houses

Working through the formulas once is the best way to understand them. Six house sales have floor area $ x $ in hundreds of square feet and price $ y $ in thousands of dollars.

Worked example

Compute the OLS line for the six houses and interpret the slope.

House	$ x_i $	$ y_i $	$ x_i – \bar x $	$ y_i – \bar y $	product	$ (x_i – \bar x)^2 $
1	13	280	−5	−70	350	25
2	15	330	−3	−20	60	9
3	17	330	−1	−20	20	1
4	18	350	0	0	0	0
5	21	380	3	30	90	9
6	24	430	6	80	480	36
Sum	108	2,100	0	0	1,000	80

Step 1Means: $ \bar x = 108/6 = 18 $, $ \bar y = 2{,}100/6 = 350 $. Both deviation columns sum to zero, a check on the arithmetic.

Step 2Slope: $ \hat\beta_1 = 1{,}000/80 = 12.5 $.

Step 3Intercept: $ \hat\beta_0 = 350 – 12.5 \times 18 = 125 $.

Step 4Line: $ \widehat{\text{price}} = 125 + 12.5\,\text{size} $.

Each extra 100 square feet is associated with a price $12,500 higher, or $125 per square foot. The intercept, $125,000 for a house of zero size, is far outside the data (the smallest house has 1,300 square feet) and has no useful meaning.

OLS by hand: six house sales price in $000

House saleOLS lineResidual

Figure 1. The teal segments are the residuals. OLS chooses the line that makes the sum of their squares, 500 here, as small as possible, and the line always passes through the point of means. Data: Econometrics Made Simple, Chapter 3, Worked Example 3.2.

Exam tipSet out the deviations table even if your calculator has a regression mode. Examiners give method marks for $ \bar x $, $ \bar y $, $ \sum (x_i – \bar x)(y_i – \bar y) $ and $ \sum (x_i – \bar x)^2 $, and the two zero-sum columns catch mistakes early. Finish with a sentence interpreting the slope in the units of the data.

OLS on 500 house sales

Now apply the same formulas to all 500 sales in houses.csv (simulated companion data, price in $000 and floor area in square feet). Econometrics Made Simple computes the estimates from the formulas and with statsmodels, and they agree:

\[ \widehat{\text{price}} = 115.22 + 0.1408\,\text{sqft}, \qquad n = 500, \quad R^2 = 0.552. \]

One more square foot is associated with a price about 0.1408 thousand dollars, or $140.8, higher; 100 more square feet with about $14,076 more. That is close to the $125 per square foot from the six houses, but based on far more data.

Price against floor area, 500 house sales price in $000

House saleOLS line

Figure 2. Bigger houses sell for more, about $141 per extra square foot, but prices vary widely at every size: floor area explains 55% of the variation. Data: houses.csv (simulated companion data), Econometrics Made Simple, Chapter 3.

Worked example

Predict the price of a 2,000 and a 1,500-square-foot house, and the difference between them.

Step 12,000 sq ft: $ 115.2163 + 0.140765 \times 2{,}000 = 396.75 $, about $396,700.

Step 21,500 sq ft: $ 115.2163 + 0.140765 \times 1{,}500 = 326.36 $, about $326,400.

Step 3Difference: $ 0.140765 \times 500 = 70.38 $, about $70,400.

You never need the intercept for a difference: it is always $ \hat\beta_1 \Delta x $.

Practise until it sticks: Econometrics Made Simple Practice Questions has over 1,150 questions, each with a worked answer.Get the practice book · $9

Fitted values, residuals and their properties

Every observation gets a fitted value $ \hat y_i = \hat\beta_0 + \hat\beta_1 x_i $ and a residual $ \hat u_i = y_i – \hat y_i $, so $ y_i = \hat y_i + \hat u_i $. A residual is not the same as the error: the error is the distance to the unknown population line and is never seen; the residual is the distance to the estimated line.

Algebraic properties of OLS (with an intercept)1. The residuals sum to zero: $ \sum \hat u_i = 0 $. 2. They are uncorrelated with $ x $: $ \sum x_i \hat u_i = 0 $. 3. The point of means $ (\bar x, \bar y) $ is on the line. Consequences: the fitted values average $ \bar y $, and they are uncorrelated with the residuals.

In the six-house example the residuals are $ -7.5, 17.5, -7.5, 0, -7.5, 5 $. They add to zero, and so does $ \sum x_i \hat u_i $. House 4 sits exactly at the means, on the line. House 2 sold for $17,500 more than the line predicts. In the 500 houses, the first house (2,705 sq ft) sold for $440,700 against a predicted $495,980, a residual of $-55.28$: about $55,000 cheap for its size. The largest residual, $+325.4$, belongs to house 112, a 2,425-square-foot house in the Central neighbourhood with a pool. Its location and pool are both in $ u $ in this model.

Common mistakeUsing $ \sum x_i \hat u_i = 0 $ as evidence that $ \text{E}(u \mid x) = 0 $ holds. The residuals are uncorrelated with $ x $ by construction, in every sample, even when the true errors are strongly related to $ x $. A zero correlation here tells you the arithmetic is right, not that the assumption is.

A plot of residuals against $ x $ is one of the most useful checks. A shapeless band around zero is good. A curve suggests the wrong functional form (Step 8); a fan shape suggests heteroskedasticity (Step 11). The house residuals show no curve but spread a little more for bigger houses.

Goodness of fit: SST, SSE, SSR and R-squared

How well does the line fit? Split the total variation in $ y $ into an explained part and a residual part.

\[ \underbrace{\sum (y_i – \bar y)^2}_{\text{SST}} = \underbrace{\sum (\hat y_i – \bar y)^2}_{\text{SSE}} + \underbrace{\sum \hat u_i^2}_{\text{SSR}}, \qquad R^2 = \frac{\text{SSE}}{\text{SST}} = 1 – \frac{\text{SSR}}{\text{SST}}. \]

The decomposition works because the cross-product term $ 2\sum \hat u_i(\hat y_i – \bar y) $ is zero by the algebraic properties above. $ R^2 $, the coefficient of determination, is the fraction of the sample variation in $ y $ explained by $ x $. In simple regression it equals the squared correlation between $ y $ and $ x $.

Worked example

Compute SST, SSE, SSR and $ R^2 $ for the six houses.

Step 1SST $ = 70^2 + 20^2 + 20^2 + 0 + 30^2 + 80^2 = 13{,}000 $.

Step 2SSR $ = 3 \times 7.5^2 + 17.5^2 + 0 + 5^2 = 168.75 + 306.25 + 25 = 500 $.

Step 3SSE $ = 13{,}000 – 500 = 12{,}500 $. Check: $ \hat\beta_1^2 \sum (x_i – \bar x)^2 = 12.5^2 \times 80 = 12{,}500 $.

Step 4$ R^2 = 12{,}500/13{,}000 = 0.9615 $.

Floor area explains about 96% of the price variation among these six houses, a value typical of tidy textbook examples, not of real data. For the 500 houses, SST $ = 7{,}167{,}333.0 $, SSE $ = 3{,}953{,}550.6 $ and SSR $ = 3{,}213{,}782.3 $, so $ R^2 = 0.5516 $; the correlation is 0.743 and $ 0.743^2 = 0.552 $.

Common mistakeMixing up the labels. Here SSE is explained and SSR is residual. Some books write ESS and RSS, and a few use SSE for the error sum of squares, the opposite of ours. In an exam, write out the sum, $ \sum \hat u_i^2 $.

What $ R^2 $ does not tell you. It says nothing about causality. A low $ R^2 $ does not make a slope wrong or useless: the wage regression in Step 1 has $ R^2 = 0.139 $ but a slope with $ t = 12.68 $, and values of 0.1 to 0.3 are normal for data on individuals. It does not check the functional form; the residual plot does. And it cannot compare models with different dependent variables, such as price and log(price). Step 6 adds a fifth warning: $ R^2 $ never falls when you add a variable, which is why the adjusted $ R^2 $ exists. Our post on R-squared vs adjusted R-squared covers both in depth.

Units of measurement and logs

Changing the units changes the numbers but not the economics. For a constant $ c > 0 $: multiplying $ y $ by $ c $ multiplies both $ \hat\beta_0 $ and $ \hat\beta_1 $ by $ c $; multiplying $ x $ by $ c $ divides $ \hat\beta_1 $ by $ c $ and leaves $ \hat\beta_0 $ alone; $ R^2 $ never changes. Measuring house prices in dollars makes the slope $140.76 per square foot; measuring floor area in hundreds of square feet makes it 14.08 thousand dollars per 100 square feet. Both give the same predictions and $ R^2 = 0.5516 $.

A straight line in levels says each extra unit of $ x $ adds the same amount to $ y $. Often a constant percentage effect is more natural. Taking natural logs keeps the model linear in the parameters, so OLS applies unchanged.

Model	Interpretation of $ \beta_1 $	Companion-data estimate
Level-level: $ y $ on $ x $	one more unit of $ x $ changes $ y $ by $ \beta_1 $ units	wage on educ: $2.25 per hour per year of school
Log-level: $ \log y $ on $ x $	one more unit of $ x $ changes $ y $ by about $ 100\beta_1 $%	log wage on educ: 0.1026, about 10.3% per year
Level-log: $ y $ on $ \log x $	a 1% rise in $ x $ changes $ y $ by about $ \beta_1/100 $ units	price on log sqft: 251.06, so 1% more floor area adds about $2,510
Log-log: $ \log y $ on $ \log x $	$ \beta_1 $ is the elasticity of $ y $ with respect to $ x $	log price on log sqft: 0.693, so 10% more floor area adds about 6.9% to price

The approximations are good for changes of a few per cent and poor for large ones. Step 8 gives the exact formulas, and our guide to interpreting log coefficients works through all four cases.

Regression through the origin

Sometimes theory says $ y = 0 $ when $ x = 0 $, and it is tempting to drop the intercept. Minimising $ \sum (y_i – b_1 x_i)^2 $ gives $ \tilde\beta_1 = \sum x_i y_i / \sum x_i^2 $, with no deviations from means. For the houses, forcing the line through the origin raises the slope from $140.8 to $197.7 per square foot, because the true intercept is clearly not zero and the line must tilt to reach the data. The residuals no longer sum to zero (they add to 6,294.12), and software reports an uncentred $ R^2 $ of 0.948 that looks far better than 0.552 but measures variation around zero. Computed the usual way, $ 1 – \text{SSR}/\text{SST} = 0.450 $: worse, as it must be. Keep the intercept unless theory gives a very strong reason.

The textbook and the practice book together: the Econometrics Made Simple bundle for $19 (was $24).Both books · $19

Practise: 6 questions

Try each one on paper first, then check the worked answer.

EasyQ1. A sample has $ \bar x = 6 $, $ \bar y = 40 $, sample covariance $ s_{xy} = 15 $ and sample variance $ s_x^2 = 5 $. Find the OLS line. Show worked answer

$ \hat\beta_1 = 15/5 = 3 $.
$ \hat\beta_0 = 40 – 3 \times 6 = 22 $.
$ \hat y = 22 + 3x $.

EasyQ2. A regression of exam score on hours of study gives $ \hat y = 50 + 4\,\text{hours} $. A student who studied 7 hours scored 71. Find the fitted value and the residual, and interpret the residual. Show worked answer

$ \hat y = 50 + 4 \times 7 = 78 $.
$ \hat u = 71 – 78 = -7 $.
The student scored 7 marks below the average predicted for someone who studied 7 hours.

MediumQ3. Five observations are $ (x, y) = (2, 4), (4, 7), (6, 8), (8, 12), (10, 14) $. (a) Compute $ \hat\beta_1 $ and $ \hat\beta_0 $. (b) Find the residuals and check they sum to zero. (c) Compute SST, SSR and $ R^2 $. Show worked answer

(a) $ \bar x = 6 $, $ \bar y = 9 $. $ x $ deviations: $-4, -2, 0, 2, 4$; $ y $ deviations: $-5, -2, -1, 3, 5$. Products: 20, 4, 0, 6, 20, sum 50. Squared $ x $ deviations sum to 40. $ \hat\beta_1 = 50/40 = 1.25 $, $ \hat\beta_0 = 9 – 1.25 \times 6 = 1.5 $.
(b) Fitted: 4, 6.5, 9, 11.5, 14. Residuals: 0, 0.5, −1, 0.5, 0; sum 0.
(c) SST $ = 25 + 4 + 1 + 9 + 25 = 64 $. SSR $ = 0 + 0.25 + 1 + 0.25 + 0 = 1.5 $. SSE $ = 62.5 $ (check: $ 1.25^2 \times 40 $). $ R^2 = 62.5/64 = 0.977 $.

MediumQ4. In wages.csv, $ \widehat{\text{wage}} = -5.7286 + 2.2517\,\text{educ} $, with the wage in dollars per hour. Rewrite it with weekly pay for a 37.5-hour week as the dependent variable, and say what happens to $ R^2 $. What would the slope be if education were measured in months? Show worked answer

Weekly pay $ = 37.5 \times $ wage, so both coefficients are multiplied by 37.5: $ \widehat{\text{weekly}} = -214.82 + 84.44\,\text{educ} $.
$ R^2 $ is unchanged (it is 0.125 either way).
Months $ = 12 \times $ years, so the hourly slope is divided by 12: $ 2.2517/12 = 0.1876 $ dollars per month of schooling. The intercept is unchanged.

MediumQ5. Interpret each slope. (a) $ \widehat{\log(\text{rent})} = 3.2 + 0.45\log(\text{income}) $. (b) $ \widehat{\text{wage}} = 5 + 3.6\log(\text{firmsize}) $. (c) $ \widehat{\log(\text{price})} = 5.8 – 0.012\,\text{age} $, age in years. Show worked answer

(a) Log-log: an elasticity of 0.45. Income 1% higher is associated with rent about 0.45% higher.
(b) Level-log: a firm 1% larger is associated with a wage about $ 3.6/100 = \$0.036 $ higher; 10% larger, about 36 cents.
(c) Log-level: each extra year of age is associated with a price about $ 100 \times 0.012 = 1.2\% $ lower.

ExamQ6. A simple regression on 40 observations has SST $ = 1{,}250 $ and SSR $ = 450 $, and the slope is negative. (a) Find SSE and $ R^2 $. [2] (b) Find the sample correlation between $ x $ and $ y $. [2] (c) A classmate says the model is useless because 36% of the variation is unexplained. Comment. [2] (d) Explain why this $ R^2 $ cannot be compared with that of a model for $ \log(y) $. [2] Show worked answer

(a) SSE $ = 1{,}250 – 450 = 800 $ [1]; $ R^2 = 800/1{,}250 = 0.64 $. [1]
(b) $ |r| = \sqrt{0.64} = 0.8 $ [1]; the correlation takes the sign of the slope, so $ r = -0.8 $. [1]
(c) $ R^2 = 0.64 $ is high for economic data [1]; and usefulness depends on the size and precision of the slope and on whether $ \text{E}(u \mid x) = 0 $ holds, not on $ R^2 $. [1]
(d) $ R^2 $ is the share of the variation in the dependent variable explained [1]; $ y $ and $ \log(y) $ have different variations, so the shares measure different things. [1]
Total: 8 marks.

Check yourself

1. The OLS slope in simple regression equals

The sample covariance of $ x $ and $ y $ over the sample variance of $ x $.

2. Which is not an algebraic property of OLS with an intercept?

The sum of squared residuals is zero only if every point lies exactly on the line.

3. Price (in $000) is re-measured in dollars. The slope on floor area

Multiplying $ y $ by $ c $ multiplies both coefficients by $ c $. $ R^2 $ is unchanged.

Your score: 0 out of 3

The short version

$ y = \beta_0 + \beta_1 x + u $: $ \beta_1 $ is the change in $ y $ for one more unit of $ x $, holding $ u $ fixed.
The key assumption is $ \text{E}(u \mid x) = 0 $: the unobserved factors average the same at every $ x $. It fails if, for example, ability raises both schooling and wages.
OLS minimises $ \sum \hat u_i^2 $: $ \hat\beta_1 = s_{xy}/s_x^2 $, $ \hat\beta_0 = \bar y – \hat\beta_1 \bar x $.
Residuals sum to zero, are uncorrelated with $ x $, and the line passes through $ (\bar x, \bar y) $, by construction.
SST = SSE + SSR and $ R^2 = \text{SSE}/\text{SST} $. It is not a test of causality or of the functional form.
Rescaling $ y $ by $ c $ multiplies both coefficients by $ c $; rescaling $ x $ divides the slope by $ c $. Log-level slopes are percentages; log-log slopes are elasticities.
Keep the intercept. Chapter 3 of Econometrics Made Simple has the full Python, R and Stata code and 21 exercises with answers.

Studying more than one subject? The Complete Book Bundle includes every Economics, Statistics, Mathematics and Econometrics book.

Model	Interpretation of \( \beta_1 \)	Companion-data estimate
Level-level: \( y \) on \( x \)	one more unit of \( x \) changes \( y \) by \( \beta_1 \) units	wage on educ: $2.25 per hour per year of school
Log-level: \( \log y \) on \( x \)	one more unit of \( x \) changes \( y \) by about \( 100\beta_1 \)%	log wage on educ: 0.1026, about 10.3% per year
Level-log: \( y \) on \( \log x \)	a 1% rise in \( x \) changes \( y \) by about \( \beta_1/100 \) units	price on log sqft: 251.06, so 1% more floor area adds about $2,510
Log-log: \( \log y \) on \( \log x \)	\( \beta_1 \) is the elasticity of \( y \) with respect to \( x \)	log price on log sqft: 0.693, so 10% more floor area adds about 6.9% to price

Step 3: Simple Regression and OLS

The simple regression model

The key assumption: zero conditional mean

Deriving the OLS estimators

OLS by hand: six houses

OLS on 500 house sales

Fitted values, residuals and their properties

Goodness of fit: SST, SSE, SSR and R-squared

Units of measurement and logs

Regression through the origin

Practise: 6 questions

Check yourself

The short version

Further reading on EconTutorials

Contact Info

Quick Links

Success!