Step 20: Inference for Regression
- Your fitted line \( \hat{y} = a + bx \) is an estimate of an unknown population line \( \mu_y = \beta_0 + \beta_1 x \). The slope \( b \) changes from sample to sample, so it has a standard error.
- The key test is \( H_0: \beta_1 = 0 \) (no linear relationship), using \( t = b \div SE(b) \) with df \( = n – 2 \). The interval is \( b \pm t^{*} \cdot SE(b) \).
- \( r^2 \) gives the share of variation explained; the residual standard error \( s_e \) gives the typical prediction error in the units of \( y \). Report both.
- Exam point: a prediction interval for one new case is always wider than a confidence interval for the mean response. Check L-I-N-E with a residual plot before trusting any p-value.
From a fitted line to a population line
A company plots ten months of advertising spend against sales (both in $000s) and fits a straight line. The slope comes out at about 2.5: each extra $1,000 of advertising goes with about $2,500 more sales. The marketing director wants to put that number in next year’s budget.
Three questions stand in the way. Is the slope real, or could ten noisy points produce a line this steep when there is no true relationship? If it is real, how precisely do we know it? And if the company spends $6,000 next month, what sales should it expect, and with what uncertainty?
Earlier, regression was purely descriptive. Inference needs one more idea: the sample line is an estimate of a true line for the whole population, written with Greek letters:
\( \beta_1 \) is the unknown true slope; \( b \) is our estimate of it from one sample. Just as \( \bar{x} \) estimates \( \mu \), \( b \) estimates \( \beta_1 \). A different ten months would give a different \( b \), so \( b \) has a sampling distribution and a standard error.
For the advertising data (\( x = 1, 2, \dots, 10 \); \( y = 14, 13, 20, 18, 26, 22, 30, 27, 34, 36 \)), the means are \( \bar{x} = 5.5 \) and \( \bar{y} = 24.0 \), with \( \sum(x-\bar{x})(y-\bar{y}) = 206.0 \) and \( \sum(x-\bar{x})^2 = 82.5 \). So \( b = 206.0 \div 82.5 = 2.497 \) and \( a = 24.0 – 2.497(5.5) = 10.267 \). The fitted line is \( \hat{y} = 10.267 + 2.497x \). The squared residuals add up to 55.62, so:
Note the divisor \( n – 2 \), not \( n \). Two things (slope and intercept) were estimated from the data, and each costs one degree of freedom.
Testing the slope
A slope of zero means a flat line: \( x \) tells you nothing about \( y \). So the central test in regression is \( H_0: \beta_1 = 0 \) against \( H_a: \beta_1 \neq 0 \). The test follows the usual pattern, (estimate − hypothesised value) ÷ standard error:
The slope is estimated more precisely when the points hug the line (small \( s_e \)) and when the \( x \)-values are spread out (large \( \sum(x-\bar{x})^2 \)).
For the advertising data, \( b = 2.497 \), \( s_e = 2.637 \), \( \sum(x-\bar{x})^2 = 82.5 \), \( n = 10 \). Test whether the slope differs from zero at \( \alpha = 0.05 \), and give a 95% interval for \( \beta_1 \).
There is very strong evidence of a positive linear relationship. We are 95% confident that each extra $1,000 of advertising goes with between about $1,830 and $3,170 of extra sales. The interval excludes zero, which matches the significant test. The honest message for the budget: the effect is real, but the range is wide, so planning on the optimistic end would be unwise.
How well does the line fit?
Two numbers summarise fit, and they answer different questions.
- \( r^2 \), the coefficient of determination, is the proportion of the variation in \( y \) explained by the linear relationship with \( x \). It has no units and lies between 0 and 1.
- \( s_e \) is the typical prediction error in the original units of \( y \). It answers the practical question: how far off is a typical prediction?
For the advertising data, \( r^2 = 0.902 \): about 90% of the variation in sales is explained by advertising, leaving 10% to things like season, competitors and pricing. And \( s_e = 2.637 \), so a typical prediction misses actual sales by about $2,640. With average monthly sales of $24,000, that may or may not be acceptable for planning. That is why you report \( s_e \) alongside \( r^2 \).
Two kinds of interval for a prediction
Suppose the company plans to spend $6,000. Putting \( x = 6 \) into the line gives \( \hat{y} = 10.267 + 2.497(6) = 25.25 \), about $25,250. But an interval around that number can mean two different things.
- A confidence interval for the mean response estimates the average sales across all months with $6,000 of advertising.
- A prediction interval estimates the sales in one particular new month with that spend.
The prediction interval is always wider. It carries two sources of uncertainty: where the true line sits, plus the scatter of individual months around that line. Averaging washes out the second source; predicting a single month does not.
At \( x = 6 \) the book’s 95% intervals are:
| Interval | Range ($000s) | Meaning |
|---|---|---|
| Mean response | (23.30, 27.20) | Average sales of all $6,000 months |
| Prediction | (18.86, 31.63) | Sales in one new $6,000 month |
The prediction interval is about three times wider. If the director asks what next month will bring, the wider interval is the honest answer. Both bands are narrowest near \( \bar{x} \) and widen towards the edges of the data, which is another reason not to extrapolate far beyond the \( x \)-values you observed.
Checking the assumptions and reading output
All of this rests on four conditions, remembered as L-I-N-E:
| Condition | How to check | |
|---|---|---|
| L | Linear relationship | Scatterplot; residual plot with no curve |
| I | Independent observations | Study design (watch for data collected over time) |
| N | Normally distributed residuals | Histogram or quantile plot of residuals |
| E | Equal variance of residuals | Residual plot with constant vertical spread |
The residual plot (residuals against \( x \) or fitted values) checks three of the four at once. A healthy plot is a shapeless band around zero. A curve means the straight-line model is wrong. A funnel means unequal variance, so standard errors and intervals are unreliable; transforming \( y \), often with a logarithm, can help. A single extreme point may be distorting the fit, so refit without it and compare. For the advertising data, L, N and E look fine, but the data is monthly, so independence is questionable and should be reported as a limitation.
Software condenses everything into one table:
| Coefficients | Estimate | Std. Error | t value | p-value |
|---|---|---|---|---|
| (Intercept) | 10.267 | 1.802 | 5.70 | 0.00045 |
| advertising | 2.497 | 0.290 | 8.60 | 0.000026 |
Residual standard error: 2.637 on 8 degrees of freedom; R-squared: 0.902. Read four things first: the slope (effect size, in units), its p-value, its interval (or the SE that builds it), and \( s_e \). The intercept’s p-value is usually of no interest. As a rough guide, \( |t| \) above about 2 suggests significance at 5%. Then look at the residual plot before believing any of it.
One unifying idea: every test in this part of the book has the same skeleton. Form an estimate, divide by its standard error, and compare with a reference distribution (t, z, \( \chi^2 \) or F). Testing a slope is the same as testing a correlation, and for simple regression the F in the output equals \( t^2 \).
Practise: 6 questions
Try each one on paper first, then check the worked answer.
EasyQ1. A slope has \( b = 4.2 \) and \( SE(b) = 1.5 \) from \( n = 20 \) points. Compute t and df. Show worked answer
df \( = n – 2 = 18 \).
EasyQ2. A regression on 27 points has residual sum of squares 200. Find \( s_e \). Separately, if \( s_e = 6 \) and \( \sum(x-\bar{x})^2 = 144 \), find \( SE(b) \). Show worked answer
\( SE(b) = 6 \div \sqrt{144} = 6 \div 12 = 0.5 \)
MediumQ3. \( b = 0.8 \), \( SE(b) = 0.5 \), \( n = 12 \). With \( t^{*} = 2.228 \), is the slope significant at 5%? Show worked answer
\( 1.60 \lt 2.228 \), so not significant (two-tailed p ≈ 0.14).
No linear relationship has been established; a slope of 0 is still plausible.
MediumQ4. Using \( \hat{y} = 10.267 + 2.497x \) (all in $000s), predict sales at \( x = 8 \). If the CEO asks what one new month at that spend will bring, which interval should you give and why? Show worked answer
Give a prediction interval, because one specific month is a single new observation, not an average. It includes the scatter of individual months around the line, so it is wider than the confidence interval for the mean.
MediumQ5. A residual plot shows (a) a clear U-shape, (b) a fan that widens as \( x \) increases. Which condition fails in each case, and what should you do? Show worked answer
(b) Equal variance fails. Standard errors, intervals and p-values become unreliable, especially at the wide end. Try transforming \( y \) (for example a logarithm) and re-check the residual plot.
ExamQ6. A study of 15 stores regresses sales on floor area: \( b = 3.4 \), \( SE(b) = 1.2 \), \( s_e = 8.5 \), \( r^2 = 0.38 \). (a) Test whether the slope differs from zero using \( t^{*} = 2.160 \) at \( \alpha = 0.05 \). (b) Build a 95% interval for \( \beta_1 \). (c) Interpret \( r^2 \) and \( s_e \) together. (d) A manager says “bigger stores cause higher sales”. Comment. [8 marks] Show worked answer
(b) \( 3.4 \pm 2.160 \times 1.2 = 3.4 \pm 2.59 = (0.81,\ 5.99) \) [2].
(c) Only 38% of the variation in sales is explained by floor area, and a typical prediction misses by about 8.5 units: the relationship is real but weak and imprecise [2].
(d) Not justified: this is observational data showing association only. Lurking variables such as location or footfall could drive both [1].
Check yourself
1. Why is df \( = n – 2 \) in a slope test?
2. Output reads “Residual standard error: 2.637 on 8 degrees of freedom”. How many observations were there?
3. Which L-I-N-E condition cannot be checked from a residual plot?
Your score: 0 out of 3
The short version
- The sample line \( \hat{y} = a + bx \) estimates the population line \( \mu_y = \beta_0 + \beta_1 x \); \( b \) has a standard error \( SE(b) = s_e / \sqrt{\sum(x-\bar{x})^2} \).
- Test \( H_0: \beta_1 = 0 \) with \( t = b/SE(b) \), df \( = n – 2 \). The interval \( b \pm t^{*} SE(b) \) excludes zero exactly when the test is significant.
- \( r^2 \) is the share of variation explained; \( s_e \) is the typical error in the units of \( y \). Report both.
- A prediction interval for one new case is always wider than a confidence interval for the mean response, and both widen away from \( \bar{x} \).
- Check L-I-N-E with the residual plot before trusting the output. The full chapter in Statistics Made Simple has 56 practice questions with a complete answer key.
Further reading on EconTutorials
Older in-depth articles on this topic from our blog: