Econometrics guide, Unit 4, step 14 of 20Previous step Next step

Step 14: Logit, Probit and Binary Outcomes

By the EconTutorials team. 14-minute read with 6 practice questions.

SyllabusUndergraduate econometrics · Wooldridge, Introductory Econometrics, Section 7.5 and Ch. 17

In 30 seconds

With a 0/1 outcome, $ E(y \mid x) = P(y = 1 \mid x) $, so any regression for $ y $ is a model for a probability.
The linear probability model (OLS on the 0/1 outcome) is easy to read, in percentage points, but can predict outside 0–1, imposes constant effects and is heteroskedastic by construction.
Logit and probit pass the index $ x\beta $ through an S-shaped CDF, so probabilities stay between 0 and 1. They are estimated by maximum likelihood.
Raw logit and probit coefficients give only sign and significance. Sizes come from predicted probabilities and marginal effects, ideally average marginal effects.
Exam point: compare the percent correctly predicted with the base rate, and use the likelihood-ratio test $ 2(\ell_{UR} – \ell_R) \sim \chi^2_q $ for joint hypotheses.

Binary outcomes are everywhere

Many important outcomes in economics are decisions, not amounts. A bank approves a loan or does not. A person is employed or is not. A school leaver enrols at university or does not. Each is a variable that takes the value 1 for “yes” and 0 for “no”, and the question is always how the probability of a yes changes when an explanatory variable changes.

Key termA binary (dummy) dependent variable equals 1 if an event happens and 0 if not. Because $ y $ is 0 or 1, $ E(y \mid x) = 1 \times P(y=1 \mid x) + 0 \times P(y = 0 \mid x) = P(y = 1 \mid x) $, the response probability.

Chapter 14 of Econometrics Made Simple covers this in full: the LPM, logit and probit, maximum likelihood by hand, marginal effects, pseudo R² and LR tests.Get the textbook · $15

The running example is the book’s loans.csv: 2,000 applications for personal loans, 72.15% of them approved. The explanatory variables are the applicant’s credit score (450–850), the log of income and of the loan amount requested (both in $000), the debt ratio (existing debt payments as a share of income) and dummies for self-employment and gender. The data were simulated from a known logit model in which gender plays no part, so we can check whether the methods find the truth.

The linear probability model

The simplest approach is to run OLS exactly as before. The population regression function is then a probability, $ P(y=1 \mid x) = \beta_0 + \beta_1x_1 + \dots + \beta_kx_k $, and each slope is the change in that probability for a one-unit increase in $ x_j $: multiplied by 100, a change in percentage points.

Worked example

An LPM on the loans data, with robust standard errors, gives coefficients of 0.00222 on credit score, −0.6775 on the debt ratio, −0.1185 on self-employment and 0.1550 on log income (Econometrics Made Simple, Chapter 14). Interpret each.

Step 1Credit score: 100 more points raise the approval probability by $ 100 \times 0.00222 = 0.222 $, or 22.2 percentage points.

Step 2Debt ratio: a rise of 0.1 (say from 25% to 35% of income) lowers it by $ 0.1 \times 0.6775 = 0.068 $, about 6.8 points.

Step 3Self-employed: 11.9 points less likely to be approved than an employee with the same credit score, income, debt and loan size.

Step 4Log income is a level-log term: 10% more income adds $ 0.1550 \times \log(1.1) = 0.0148 $, about 1.5 points.

All four are highly significant with robust standard errors. The loan-amount coefficient (−0.0526) is significant only at 10%.

Common mistakeReading −0.1185 as “an 11.85% lower probability”. A per cent of a probability is ambiguous: 11.85% of 0.72 is only 0.085. The coefficient is a change in the probability itself, so say percentage points: from, say, 0.74 to 0.62.

Three problems with the LPM

1. Predictions outside 0 and 1. A straight line keeps going. In the loans data the fitted values run from −0.057 to 1.324: 1 applicant has a negative “probability” and 153 have probabilities above 1, so 7.7% of the sample get impossible predictions.

2. Constant marginal effects. The LPM says 50 extra credit points add 11.1 percentage points whether an applicant starts at 0.3 or 0.95. Near the top that is impossible: effects must shrink as the probability approaches 0 or 1.

3. Built-in heteroskedasticity. Given $ x $, $ y $ is a Bernoulli variable with variance $ p(1-p) $, so

\[ \text{Var}(u \mid x) = p(x)\,[1 – p(x)], \]

which changes with $ x $. The usual standard errors are wrong, and the fix is the robust standard errors of Step 11. Even so, the LPM is not dead: many labour and development economists report it as their main result, because its coefficients are average effects anyone can read, and it handles fixed effects and IV easily. The usual practice is to show the LPM and confirm that a logit or probit tells the same story.

Logit and probit

The way out is to keep the index $ x\beta = \beta_0 + \beta_1x_1 + \dots + \beta_kx_k $ but pass it through a function $ G $ that always lies between 0 and 1: $ P(y = 1 \mid x) = G(x\beta) $. The natural choice is a cumulative distribution function.

\[ \text{Logit: } G(z) = \Lambda(z) = \frac{e^{z}}{1+e^{z}} = \frac{1}{1+e^{-z}}, \qquad \text{Probit: } G(z) = \Phi(z) = \int_{-\infty}^{z} \frac{1}{\sqrt{2\pi}}e^{-v^2/2}\,dv . \]

Both equal 0.5 at $ z = 0 $, are steepest there and flatten towards 0 and 1. That flattening is what the LPM lacks: a change in $ x\beta $ moves the probability a lot near 0.5 and hardly at all near 0 or 1.

Both models come from a simple story. Suppose the bank computes an unobserved net benefit of lending, $ y^* = x\beta + e $, and approves when $ y^* > 0 $. Then $ P(y = 1 \mid x) = P(e > -x\beta) = G(x\beta) $ when $ e $ is symmetric with CDF $ G $: logistic errors give logit, standard normal errors give probit. Because the scale of $ y^* $ is never observed, the error variance is fixed by convention: 1 for probit and $ \pi^2/3 \approx 3.29 $ for logit. That is why logit coefficients are larger than probit ones, and why neither is a change in probability.

Key termA latent variable $ y^* $ is an unobserved continuous variable whose sign determines the observed binary outcome. The coefficients measure effects on $ y^* $ (for logit, on the log of the odds). Their sign and significance carry over to the probability; their size does not.

Maximum likelihood

OLS minimises the sum of squared residuals. Logit and probit use a different principle: maximum likelihood (ML) chooses the parameters that make the observed outcomes most likely. An approved applicant contributes $ G(x_i\beta) $ to the likelihood; a rejected one contributes $ 1 – G(x_i\beta) $. With independent observations we multiply, and taking logs turns the product into a sum:

\[ \ell(\beta) = \sum_{i=1}^{n} \Big\{ y_i \log G(x_i\beta) + (1 – y_i) \log\big[1 – G(x_i\beta)\big] \Big\}. \]

Each term is the log of a probability, so $ \ell $ is negative and gets closer to zero as the fit improves. There is no formula for the maximum; the computer climbs the hill step by step (Newton–Raphson). In the book’s hand-coded version for the loans logit, the log-likelihood starts at $ 2{,}000 \times \log 0.5 = -1{,}386.29 $, jumps to −1,001.51 after one step and settles at −974.11 after four. ML estimators are consistent, approximately normal and efficient in large samples, which is why the output reports $ z $ rather than $ t $ statistics.

Worked example

Three applicants have outcomes $ y = (1, 0, 1) $ and, at a trial value of $ \beta $, predicted approval probabilities 0.8, 0.3 and 0.6. Compute the log-likelihood. Is a value of $ \beta $ giving 0.9, 0.2 and 0.7 better?

Step 1$ \ell = \log 0.8 + \log(1-0.3) + \log 0.6 = -0.2231 – 0.3567 – 0.5108 = -1.0906 $.

Step 2Second value: $ \log 0.9 + \log 0.8 + \log 0.7 = -0.1054 – 0.2231 – 0.3567 = -0.6852 $.

The second is closer to zero, so it makes the observed outcomes more likely. ML keeps moving $ \beta $ this way until no further improvement is possible.

The logit estimates are: intercept −9.2401, credit score 0.013808, log income 0.9611, debt ratio −4.0723, self-employed −0.6440 and log loan amount −0.2993. All but the loan amount ($ z = -1.63 $, $ p = 0.102 $) have $ |z| > 4 $. The probit coefficients are smaller by a factor between 1.67 and 1.73 for every variable.

Interpreting the results

Predicted probabilities

To get a probability, compute the index for a particular applicant and pass it through $ G $.

Worked example

An employee with a credit score of 700 and income of $50,000 asks to borrow $150,000; her debt ratio is 0.30. Use the logit to find her approval probability, then with 50 more credit points, and if she were self-employed.

Step 1With $ \log 50 = 3.9120 $ and $ \log 150 = 5.0106 $: $ x\hat\beta = -9.2401 + 0.013808(700) + 0.961145(3.9120) – 4.072255(0.30) – 0.299331(5.0106) = 1.4640 $.

Step 2$ \Lambda(1.4640) = 1/(1+e^{-1.4640}) = 0.8121 $: about an 81% chance.

Step 3Score 750: the index rises by $ 0.013808 \times 50 = 0.6904 $ to 2.1544, and $ \Lambda(2.1544) = 0.8961 $: up 8.4 points.

Step 4Self-employed: the index falls by 0.6440 to 0.8200, and $ \Lambda(0.8200) = 0.6942 $: down 11.8 points.

The LPM would give the same 11.1 points for 50 credit points to everybody. The logit gives her less because she is already high up the S-curve, where it flattens.

Predicted approval probability against credit score loans.csv, n = 2,000

linear probability modellogitprobit

Figure 1. Other variables at their sample means and self_employed = 0. In the middle of the data the three models agree; the LPM line keeps rising and passes 1 at a credit score of about 800, while the logit and probit curves, almost on top of each other, flatten towards 1. Recomputed from the loans data in Econometrics Made Simple, Chapter 14.

Marginal effects

For a continuous variable, the effect of a small change in $ x_j $ on the probability follows from the chain rule:

\[ \frac{\partial P(y=1 \mid x)}{\partial x_j} = g(x\beta)\,\beta_j, \qquad \text{logit: } p(1-p)\,\beta_j, \qquad \text{probit: } \phi(x\beta)\,\beta_j . \]

The scale factor is largest at $ p = 0.5 $: 0.25 for logit and 0.399 for probit. Since it depends on $ x $, every applicant has a different marginal effect, and you must decide where to evaluate it. The marginal effect at the mean (MEM) evaluates it for an “average applicant”; the average marginal effect (AME) computes it for every observation and averages.

In the loans logit the AME of credit score is 0.00221 (2.2 points per 10 credit points): the average of $ \hat{p}_i(1-\hat{p}_i) $ is 0.1599, and $ 0.1599 \times 0.013808 = 0.00221 $. The MEM is 0.00244, about 10% larger, because at the means $ \hat{p} = 0.7703 $ and $ 0.7703 \times 0.2297 = 0.1769 $. Many applicants have probabilities near 1, where $ p(1-p) $ is small, and they pull the AME down. The MEM also describes someone who does not exist: at the means, self-employment equals 0.12.

For a dummy, use the discrete change $ G(x\beta \mid d=1) – G(x\beta \mid d=0) $, averaged over the sample. For self-employment it is −0.111: about 11.1 points lower on average. In a logit, $ e^{\beta_j} $ is also the odds ratio: $ e^{-0.644} = 0.525 $, so self-employment roughly halves the odds of approval.

Exam tipPrefer the AME and say which one you report. It answers the policy question “if every applicant’s credit score rose by one point, how much would the approval rate rise?”. In Stata, margins, dydx(*) gives AMEs (add atmeans for MEMs); in statsmodels, get_margeff(), with dummy=True for discrete changes.

LPM, logit and probit side by side

Average effect on P(approved)	LPM	Logit AME	Probit AME
Credit score (per point)	0.0022	0.0022	0.0022
Log income	0.1550	0.1537	0.1534
Debt ratio	−0.6775	−0.6511	−0.6546
Self-employed	−0.1185	−0.1110	−0.1126
Log loan amount	−0.0526	−0.0479	−0.0492

The agreement is striking. For average effects in a sample whose probabilities sit mostly inside (0, 1), the choice of model hardly matters; where the models differ is in predictions for unusual applicants, as Figure 1 shows. Rules of thumb for converting raw coefficients: $ \hat\beta_{\text{logit}} \approx 1.6\,\hat\beta_{\text{probit}} $, and near $ p = 0.5 $, $ \hat\beta_{\text{LPM}} \approx 0.25\,\hat\beta_{\text{logit}} \approx 0.4\,\hat\beta_{\text{probit}} $. Our article Logit vs Probit goes through the choice in more depth.

Goodness of fit

The usual $ R^2 $ is built on sums of squares, which ML does not minimise. The basic measure is the maximised log-likelihood: −974.11 for logit, −974.20 for probit. Compare it with a model containing only an intercept, which predicts 0.7215 for everyone and has $ \ell_0 = -1{,}183.06 $:

\[ \text{McFadden pseudo } R^2 = 1 – \frac{\ell_{UR}}{\ell_0} = 1 – \frac{-974.11}{-1{,}183.06} = 0.1766 . \]

Pseudo $ R^2 $ values are usually much lower than OLS $ R^2 $ values for equally good models; they are most useful for comparing specifications on the same data.

A more intuitive measure predicts approval when $ \hat{p} > 0.5 $ and counts the hits. The logit gets 208 + 1,323 = 1,531 of 2,000 right: 76.55%. But a “model” that approves everyone would score 72.15%, the base rate, and the logit gets 91.7% of approvals right but only 37.3% of rejections (208 of 557). Always report the base rate and the success rate for each outcome.

Hypothesis tests and a question of fairness

For one coefficient, the $ z $ test on the raw coefficient is all you need: because $ G $ is increasing, $ \beta_j = 0 $ exactly when $ x_j $ has no effect on the probability. For several restrictions, use the likelihood ratio:

\[ LR = 2\,(\ell_{UR} – \ell_{R}) \;\sim\; \chi^2_q \ \text{ approximately, under } H_0. \]

All slopes zero: $ LR = 2(-974.11 + 1{,}183.06) = 417.90 $ on 5 degrees of freedom, a massive rejection.

Does the bank treat women differently? A raw comparison is not enough, because men and women may differ in credit score or debt. Add female to the logit and test it, holding the credit factors fixed. Raw approval rates are already similar (72.4% for men, 71.9% for women). The coefficient on female is 0.030 with $ z = 0.27 $; the LR test of female alone gives 0.073 ($ p = 0.787 $), and its AME is half a percentage point. A joint test of self-employment and gender gives $ LR = 15.749 $ ($ p = 0.00038 $), but that rejection is driven entirely by self-employment. No evidence of discrimination: the right answer, since gender played no part in the simulation.

Common mistakeTreating an insignificant gender coefficient as proof of fairness. If credit scores are themselves shaped by past discrimination, holding them fixed hides it; and if a factor the lender uses is omitted and differs by gender, the coefficient picks it up. Report a precise zero with its confidence interval, not as a verdict.

Logit and probit start a family: multinomial logit for unordered choices, ordered probit for ratings, Poisson regression for counts, and Tobit and Heckman models for censored and selected outcomes. All are estimated by ML and read through predicted probabilities and marginal effects.

Turn logit output into probabilities without hesitation: the Practice Questions book has worked answers to every question.1,150+ questions · $9

Practise: 6 questions

Try each one on paper first, then check the worked answer.

EasyQ1. An LPM for employment gives a coefficient of 0.035 on years of education. Interpret it, and give the predicted change for four extra years. Show worked answer

Each extra year of education raises the probability of employment by 0.035, or 3.5 percentage points, holding the other variables fixed.
Four years: $ 4 \times 0.035 = 0.14 $, or 14 percentage points (the LPM assumes this is the same at every starting probability).

EasyQ2. For one person the index is $ x\hat\beta = 0.4 $. Compute the predicted probability if the model is logit, and if it is probit ($ \Phi(0.4) = 0.6554 $). What is the logit probability when the index is −0.4? Show worked answer

Logit: $ \Lambda(0.4) = 1/(1+e^{-0.4}) = 1/1.6703 = 0.599 $.
Probit: $ \Phi(0.4) = 0.655 $.
By symmetry, $ \Lambda(-0.4) = 1 – 0.599 = 0.401 $.
(The same index gives different probabilities because the two models scale their coefficients differently.)

MediumQ3. Four observations have $ y = (1, 1, 0, 0) $ and predicted probabilities $ (0.9, 0.6, 0.3, 0.2) $. Compute the log-likelihood. Which observation fits worst? Show worked answer

Contributions: $ \log 0.9 = -0.1054 $; $ \log 0.6 = -0.5108 $; $ \log(1-0.3) = -0.3567 $; $ \log(1-0.2) = -0.2231 $.
$ \ell = -1.1960 $.
The second observation fits worst: it was a 1 but the model gave it only 0.6, so its contribution is furthest from zero.

MediumQ4. A logit coefficient on a continuous variable is 0.8. Compute the marginal effect for a person with $ \hat{p} = 0.7 $ and for one with $ \hat{p} = 0.95 $, and the odds ratio. Show worked answer

At $ \hat{p} = 0.7 $: $ 0.7 \times 0.3 \times 0.8 = 0.168 $, about 16.8 points per unit.
At $ \hat{p} = 0.95 $: $ 0.95 \times 0.05 \times 0.8 = 0.038 $, only 3.8 points: the curve has flattened.
Odds ratio: $ e^{0.8} = 2.23 $: a one-unit rise multiplies the odds $ p/(1-p) $ by 2.23 for everyone.

MediumQ5. A probit has $ \ell_{UR} = -512.4 $. Dropping two regressors gives $ \ell_R = -519.8 $, and the intercept-only model has $ \ell_0 = -610.0 $. Test whether the two regressors are jointly zero at 5%, and compute McFadden’s pseudo $ R^2 $ for the full model. Show worked answer

$ LR = 2(-512.4 + 519.8) = 14.8 $, against the $ \chi^2_2 $ critical value 5.99: reject ($ p = 0.0006 $); the two regressors are jointly significant.
Pseudo $ R^2 = 1 – (-512.4)/(-610.0) = 1 – 0.840 = 0.160 $.

ExamQ6. A logit for whether 1,000 customers renew a subscription (800 renewed) gives this classification at a 0.5 cut-off: of the renewers, 760 predicted to renew and 40 not; of the non-renewers, 150 predicted to renew and 50 not. (a) Compute the percent correctly predicted and compare it with a sensible benchmark. [3] (b) Compute the share of non-renewers correctly identified, and comment. [2] (c) Suggest how the firm could identify more non-renewers, and the cost of doing so. [2] Show worked answer

(a) Correct: $ 760 + 50 = 810 $ of 1,000, 81%. [M1 A1] Benchmark: predicting “renew” for everyone is right 80% of the time, so the model adds only 1 point. [B1]
(b) $ 50/200 = 25\% $ of non-renewers identified (against 95% of renewers). [A1] If the firm wants to target likely leavers, the model is weak despite its headline 81%. [B1]
(c) Lower the cut-off below which a customer is classed as a non-renewer, for example to the sample share 0.8. [B1] More leavers are caught, but more loyal customers are wrongly flagged: one kind of error is traded for the other. [B1]
Total: 7 marks.

Check yourself

1. An LPM coefficient of −0.12 on a dummy means the probability

LPM coefficients are changes in the probability itself, so they are percentage points.

2. Logit coefficients are typically larger than probit coefficients by a factor of about

The logistic error has a larger standard deviation (1.81) than the standard normal (1); at the centre, 0.25 is about 1/1.6 of 0.399.

3. The average marginal effect (AME) is

AME = $ \tfrac{1}{n}\sum g(x_i\hat\beta)\hat\beta_j $. The MEM evaluates at the means instead.

Your score: 0 out of 3

The short version

With a 0/1 outcome, $ E(y \mid x) = P(y = 1 \mid x) $: a regression models a probability.
LPM: OLS on the 0/1 outcome, slopes in percentage points. It predicts outside [0, 1] (154 of 2,000 loans), imposes constant effects and has $ \text{Var}(u \mid x) = p(1-p) $, so use robust standard errors.
Logit and probit: $ P(y = 1 \mid x) = G(x\beta) $, from a latent $ y^* = x\beta + e $. Estimated by ML.
Raw coefficients give sign and significance; sizes come from predicted probabilities and marginal effects $ g(x\beta)\beta_j $. Prefer AMEs; use discrete changes for dummies.
Fit: log-likelihood, McFadden pseudo $ R^2 $, and the classification table compared with the base rate. Joint tests: $ LR = 2(\ell_{UR} – \ell_R) \sim \chi^2_q $.
The full chapter in Econometrics Made Simple codes ML by hand and has the code in Python, R and Stata, with a complete set of exercises and answers.

Textbook and practice questions together: get both Econometrics Made Simple books for $19 (was $24).Both books · $19

Studying more than one subject? The Complete Book Bundle gives you every Economics, Statistics, Mathematics and Econometrics book in one download.

Step 14: Logit, Probit and Binary Outcomes

Binary outcomes are everywhere

The linear probability model

Three problems with the LPM

Logit and probit

Maximum likelihood

Interpreting the results

Predicted probabilities

Marginal effects

LPM, logit and probit side by side

Goodness of fit

Hypothesis tests and a question of fairness

Practise: 6 questions

Check yourself

The short version

Further reading on EconTutorials

Contact Info

Quick Links

Success!