Logistic Regression: Predicting Probabilities for a Binary Outcome

Linear regression predicts a continuous number — a price, a score, a temperature. But what if the outcome is binary: will a customer churn or not, will a loan default or not, will a patient recover or not? Fitting an ordinary regression line to a yes/no outcome breaks down almost immediately — predictions can fall below 0 or above 1, which makes no sense as a probability. Logistic regression is built specifically to handle this case correctly.

Why Linear Regression Doesn't Work for Binary Outcomes

Logistic regression extends the same regression framework covered in simple linear regression to the extremely common case of a binary outcome — the underlying least-squares-style estimation logic changes (logistic regression uses a related but distinct method called maximum likelihood estimation), but the goal of modelling a relationship between predictors and an outcome remains the same.

Coding "did not churn" as 0 and "churned" as 1, then fitting an ordinary least-squares line, produces two serious problems: the fitted line can predict values below 0 or above 1 (meaningless as a probability), and the constant-variance assumption ordinary regression relies on is violated by the very nature of binary data. Logistic regression solves both issues by modelling something different: not the raw outcome, but the log-odds of the outcome, transformed through a function specifically designed to stay bounded between 0 and 1.

The Logistic Function

This S-shaped ("sigmoid") curve is the core of logistic regression. As the linear combination (b₀+b₁x) becomes very large and positive, the predicted probability approaches 1; as it becomes very large and negative, the predicted probability approaches 0 — but it can never actually reach exactly 0 or 1, and it never exceeds those bounds, exactly the property ordinary linear regression lacked.

Odds and Log-Odds

Logistic regression is actually linear — just in the log-odds (also called "logit") of the outcome, not the probability itself:

Odds express likelihood as a ratio rather than a proportion: odds of 3:1 mean an event is three times as likely to happen as not. Odds and probability are related but different — a probability of 0.75 corresponds to odds of 0.75/0.25 = 3, or "3 to 1."

Interpreting Coefficients as Odds Ratios

A logistic regression coefficient, once exponentiated (eᵇ), gives an odds ratio — how much the odds of the outcome multiply for each one-unit increase in that predictor, holding other predictors constant (exactly the same "holding constant" logic as multiple linear regression).

Worked example: a model predicting loan default finds a coefficient of 0.405 on "number of missed payments in the past year." e⁰·⁴⁰⁵ ≈ 1.5 — each additional missed payment is associated with a 50% increase in the odds of default (odds ratio of 1.5), holding other factors constant. An odds ratio above 1 means the predictor increases the odds of the outcome; below 1 means it decreases them; exactly 1 means no association.

From Probability to Classification: The Decision Threshold

Logistic regression outputs a probability, not a direct yes/no classification — converting that probability into an actual decision requires choosing a threshold, conventionally 0.5 (predict "yes" if P ≥ 0.5). This threshold isn't mathematically fixed, though — it can and should be adjusted based on the relative costs of different error types. For a rare, serious disease screening test, a lower threshold (flagging more borderline cases as positive) might be preferred to catch more true cases, accepting more false positives in exchange for fewer missed true positives — a trade-off decision that depends on context, not a universal default.

Evaluating a Logistic Regression Model

Since logistic regression predicts categories rather than continuous values, R² isn't the natural fit metric it is for linear regression. Instead, models are typically evaluated using accuracy, precision, recall, and the confusion matrix — metrics that directly measure classification performance against the chosen threshold, plus threshold-independent measures like the AUC-ROC curve, which summarizes how well the model separates the two classes across every possible threshold simultaneously.

Real-World Uses

Credit scoring: predicting the probability a loan applicant will default is one of logistic regression's most established real-world applications, still widely used alongside more complex modern machine learning methods for its interpretability.

Medical diagnosis: predicting the probability a patient has a specific condition based on symptoms and test results, informing diagnostic and treatment decisions.

Marketing: predicting the probability a customer will respond to a campaign, click an ad, or churn — directly informing targeting decisions.

Election forecasting: predicting the probability a specific candidate wins a given constituency, often as one component within a larger forecasting model.

Common Mistakes

Using ordinary linear regression on a binary outcome anyway. Sometimes called the "linear probability model," this is occasionally used for its simplicity, but it retains the fundamental problems covered above (predictions outside [0,1], violated variance assumptions) and is generally considered inferior to logistic regression for binary outcomes, despite superficially simpler coefficient interpretation.

Interpreting an odds ratio as a straightforward percentage change in probability. As the callout above explains, this approximation only holds reasonably well when the baseline probability is small — for outcomes closer to 50-50, the actual probability change from a given odds ratio depends on the baseline probability itself and can differ substantially from the odds ratio's apparent percentage.

Choosing a 0.5 classification threshold by default without considering the actual costs involved. As covered above, the "right" threshold depends on the relative cost of false positives versus false negatives in the specific application — a fraud-detection model and a spam filter might reasonably use very different thresholds even with identical underlying probability estimates, because the cost of each type of error differs between those two contexts.

A Worked Example: From Coefficient to Prediction

Problem: A logistic regression model for customer churn has intercept b₀=−2.0 and coefficient b₁=0.5 on "months since last purchase." What's the predicted probability of churn for a customer with 6 months since last purchase?

Linear combination: −2.0 + 0.5(6) = −2.0 + 3.0 = 1.0

P(churn) = 1/(1+e⁻¹·⁰) = 1/(1+0.368) = 1/1.368 ≈ 0.731, or about 73.1%

At the default 0.5 threshold, this customer would be classified as likely to churn. Note how the underlying linear combination (1.0) itself isn't a probability — it's the log-odds, which the logistic function then transforms into the bounded 0-to-1 probability shown here, exactly the mechanism that keeps every prediction valid regardless of how extreme the linear combination becomes.