Step 22: A First Look at Multiple Regression

The study that reversed itself

A health researcher records weekly exercise hours and blood cholesterol for twelve patients and fits a simple regression. The slope is +3.59, with p = 0.001. Taken at face value, each extra hour of weekly exercise goes with cholesterol about 3.6 points higher. Exercise looks harmful.

Then the researcher adds one more variable to the model: age. The exercise coefficient becomes −4.56, with p = 0.026. Still significant, but now firmly negative. Exercise lowers cholesterol after all.

Nothing about the data changed. Same twelve patients, same measurements. The only change was which variables the model was allowed to see. This step explains how that is possible.

Real outcomes rarely have one cause. Cholesterol depends on age, diet, genes and exercise together; house prices depend on size, location and age together. Multiple regression extends the simple regression of Step 15 to several predictors:

Each coefficient still has a standard error, a t-statistic, a p-value and a confidence interval. The residual standard error still measures typical prediction error, and the L-I-N-E conditions still apply. What changes, and changes a lot, is what a coefficient means.

What “controlling for” means

“Controlling for age” sounds technical, but the idea is simple: compare people of similar age. Instead of asking “do people who exercise more have higher cholesterol?”, you ask “among people of the same age, do those who exercise more have higher cholesterol?” These are different questions, and they can have different answers.

In the researcher’s sample, the people who exercised most were also the oldest: the correlation between age and exercise is 0.97. Age strongly raises cholesterol, and that alone pushes the whole cloud of points upward as exercise increases. But within any age band, more exercise goes with lower cholesterol. The overall upward trend was age in disguise.

This is the lurking variable from earlier in the book, now with a remedy. Before, you could only warn that a third variable might be driving things. Multiple regression lets you put that variable in the model and ask the question you actually care about.

The case study resolved: reading two models

Here are the two models for the same 12 patients.

Model 1 has intercept 188.58, \( s_e = 5.72 \) on 10 df and \( R^2 = 0.673 \). Model 2 has intercept 147.23, \( s_e = 3.14 \) on 9 df, \( R^2 = 0.911 \) and adjusted \( R^2 = 0.892 \).

Reading Model 2. The exercise coefficient says: among patients of the same age, each extra weekly hour of exercise goes with cholesterol about 4.6 points lower. The age coefficient says: among patients who exercise the same amount, each extra year of age goes with cholesterol about 1.6 points higher.

Which model is right? Both are correctly calculated, but Model 2 answers the sensible question. Model 1 lumps 30-year-olds who barely exercise together with 65-year-olds who exercise daily, so its slope blends an exercise effect with an age effect. Model 2 separates them, explains more (\( R^2 \) up from 0.67 to 0.91) and predicts better (\( s_e \) down from 5.72 to 3.14).

Adjusted \( r^2 \) and overlapping predictors

Adding a predictor can never reduce plain \( r^2 \). Even a column of random numbers nudges it up a little, because the extra flexibility lets the line fit the sample more closely. So plain \( r^2 \) is useless for deciding whether a predictor is worth including.

Adjusted \( r^2 \) fixes this by charging a penalty for each extra predictor. It rises only when a new variable adds more than the noise it brings, and falls otherwise.

Adding age was clearly worthwhile: both measures jumped. Adding shoe size was not: plain \( R^2 \) crept up (as it always must), but adjusted \( R^2 \) fell from 0.892 to 0.880. Model 2 is preferred. Plain \( R^2 \) alone would have wrongly picked Model 3.

Throwing every available variable into a model is a reliable way to produce nonsense. Some predictors will look significant by chance (the multiple-comparisons problem from Step 14), the model will fit the sample’s quirks rather than the population (overfitting), and coefficients become unstable. Choose predictors because you have a reason to expect them to matter.

The cholesterol data shows this too. Age and exercise correlate at 0.97, so they carry almost the same information. When age entered the model, the exercise coefficient’s standard error more than doubled, from 0.791 to 1.711. The model as a whole improved, but that individual estimate became less precise. Multicollinearity harms the interpretation of individual coefficients, not the model’s predictions. Report it, interpret individual coefficients with care, and if two predictors are near-duplicates, consider keeping only one.

Categorical predictors, and where statistics goes next

Many important variables are categorical: smoker or non-smoker, three regions, four store types. Regression handles them with a dummy variable, coded 1 if a case is in the category and 0 if not. The category coded 0 is the reference group.

Suppose a fitted model is \( \hat{y} = 145.0 – 4.2(\text{exercise}) + 1.5(\text{age}) + 9.3(\text{smoker}) \). The smoker coefficient means: comparing a smoker and a non-smoker of the same age who exercise the same amount, the smoker’s cholesterol is about 9.3 points higher. For a 45-year-old smoker who exercises 3 hours a week, \( \hat{y} = 145.0 – 12.6 + 67.5 + 9.3 = 209.2 \). An otherwise identical non-smoker gets 199.9, exactly 9.3 lower.

A category with more than two levels needs one fewer dummy than the number of categories. Three regions need two dummies, and the left-out region is the reference group. Using a dummy for every category would make the dummies add up to 1 for every case, which is perfect multicollinearity, and the model could not be solved.

With dummies, regression quietly contains earlier methods. A regression on one two-category dummy reproduces the two-sample t-test; a set of dummies for several groups reproduces one-way ANOVA. That generality is why regression dominates applied statistics.

Beyond a first course, the same estimate-and-standard-error logic extends to:

• Logistic regression for yes/no outcomes (will a customer leave?). Ordinary regression could predict probabilities below 0 or above 1, so logistic regression models the odds instead.
• Time series methods for ordered data where neighbours are correlated, such as forecasting sales.
• Causal inference: randomised experiments remain the gold standard, and methods such as instrumental variables, difference-in-differences and regression discontinuity draw causal claims from observational data.
• Machine learning, which puts prediction ahead of explanation.

Every one of these can be used to mislead. p-hacking (testing many things and reporting only what reached significance) manufactures false findings; the remedy is to decide the analysis in advance and report everything you tried. Models learn the biases in their training data, so “the algorithm decided” is no answer to a question about fairness. And “anonymised” data can often be re-identified by combining it with other datasets. The habit worth keeping is one question: what would have to be true for this conclusion to be wrong?

Practise: 6 questions

Try each one on paper first, then check the worked answer.

Check yourself

The short version

• Multiple regression \( \hat{y} = b_0 + b_1x_1 + b_2x_2 + \dots \) extends simple regression to several predictors, with the same inference tools.
• Each coefficient means “holding the others constant”, so it has no meaning apart from its model. Controlling for a confounder can change a result’s size, significance or sign, as the flip from +3.59 to −4.56 shows.
• Judge new predictors with adjusted \( r^2 \). Multicollinearity inflates standard errors and makes individual coefficients unstable.
• Categorical predictors enter as dummies (one fewer than the number of categories); each coefficient is the difference from the reference group.
• Avoid p-hacking, remember models inherit biases, and treat privacy seriously. The full chapter in Statistics Made Simple has 44 practice questions with a complete answer key, plus an appendix showing every result in R and Stata.