What is Regression Analysis ?

What is Regression Analysis?

Regression analysis is a statistical technique used to examine and quantify the relationship between variables. In economics and econometrics, it allows us to move beyond describing a relationship in words and instead estimate it precisely: by how much does the dependent variable change when an independent variable changes by one unit?

Regression Analysis is a statistical technique that explains the change in a dependent variable due to movement in one or more independent variables. It is a technique for predicting the unknown value of a variable based on the known values of other variables.

Dependent and Independent Variables

Every regression model has:

• Dependent variable (Y) — the variable we are trying to explain or predict. Also called the regressand, explained variable, or response variable.
• Independent variable(s) (X) — the variables we use to explain Y. Also called regressors, explanatory variables, or predictor variables.

Example: Suppose we want to explain the quantity demanded of a good:

Q = f(P, P_s, Y_d)

Where Q (quantity demanded) is the dependent variable, and P (price), P_s (price of substitutes), and Y_d (income) are the independent variables. Changes in P, P_s, or Y_d cause changes in Q.

Important distinction: Regression analysis measures the strength and direction of a relationship, and tests whether it is statistically significant. It does not automatically prove causation — a strong statistical relationship could still be spurious if the theory underlying it is weak.

The Simple Linear Regression Model

The simple regression model involves just one independent variable and takes the form:

Y = β₀ + β₁X + ε

Where:

• Y = the dependent variable
• X = the independent variable
• β₀ = the intercept — the expected value of Y when X = 0
• β₁ = the slope coefficient — the change in Y for a one-unit increase in X (ceteris paribus)
• ε (epsilon) = the error term — captures all other factors affecting Y that are not included in the model

Why is it called “linear”?

The model is linear in two senses:

1. If plotted, the relationship between Y and X is a straight line.
2. It is linear in the parameters (β₀ and β₁) — this is the more technically precise meaning in econometrics.

Chapter 3 of Econometrics Made Simple fits a simple regression by hand and then on a house-price dataset, step by step.Get the textbook · $15

The Multiple Regression Model

Most economic phenomena are influenced by more than one variable. The multiple regression model extends the simple model to include several independent variables:

Y = β₀ + β₁X₁ + β₂X₂ + … + β_kX_k + ε

Each coefficient β_i measures the effect of X_i on Y, holding all other variables constant (ceteris paribus). This is the power of multiple regression — it allows us to isolate the individual effect of each variable.

Example: Wage Equation

Wage = β₀ + β₁(Education) + β₂(Experience) + β₃(Female) + ε

Here β₁ tells us how much wages increase with an extra year of education, holding experience and gender constant. β₃ on the Female dummy variable measures the wage gap between female and male workers with equal education and experience.

Ordinary Least Squares (OLS) Estimation

The most common method for estimating regression coefficients is Ordinary Least Squares (OLS). OLS finds the values of β₀ and β₁ that minimise the sum of squared residuals — the sum of squared differences between observed Y values and the values predicted by the regression line.

Minimise: Σ(Y_i − Ŷ_i)²

Where Ŷ_i is the predicted value of Y for observation i. OLS is the Best Linear Unbiased Estimator (BLUE) when the classical regression assumptions are satisfied (Gauss-Markov theorem).

Make regression second nature: the Practice Questions book starts with the basics and every question has a worked answer.Get 1,150+ questions · $9

Key Regression Output Explained

Classical Assumptions of OLS

For OLS estimates to be unbiased and efficient, and for the usual tests to be valid, several assumptions must hold:

1. The relationship between Y and X is linear in parameters
2. No perfect multicollinearity among independent variables
3. The error term has zero conditional mean: E(ε | X) = 0 — the factors left in the error are unrelated to the regressors. This is the assumption that makes OLS unbiased; omitted variables that are correlated with X break it
4. Homoscedasticity: constant variance of the error term across observations
5. No autocorrelation: error terms are not correlated with each other
6. The error term is normally distributed (needed for exact t and F tests in small samples; in large samples the tests are approximately valid without it)

Violations of these assumptions (e.g., heteroskedasticity, autocorrelation, endogeneity) require corrective techniques such as robust standard errors or instrumental variables. High but imperfect multicollinearity is not a violation: OLS stays unbiased, but standard errors get larger.

Summary

Regression analysis is the cornerstone of econometrics. It allows economists to estimate the quantitative relationship between variables, test economic theories with data, and make predictions. The simple model Y = β₀ + β₁X + ε provides the foundation, while the multiple regression model extends it to control for many factors simultaneously. OLS is the standard estimation method, and understanding its assumptions and output is essential for any applied economist.

Studying more than one subject? The Complete Book Bundle gives you every Economics, Statistics, Mathematics, Econometrics and Research Methods book from EconTutorials in one download.

Textbook plus 1,150+ practice questions: get both Econometrics Made Simple books for $19 (was $24).Both books · $19