Maximum Likelihood Estimation: The Method Behind the Formulas

Logistic regression and Poisson regression both mention that their coefficients are fit using "maximum likelihood estimation" rather than the ordinary least-squares method used in linear regression — but neither explains what that actually means. Maximum likelihood estimation (MLE) is a general-purpose method for finding the parameter value that makes the data you actually observed as probable as possible, and it underlies far more of applied statistics than just these two regression techniques.

The Core Idea

Given some observed data and a model with an unknown parameter, the likelihood function asks: for any candidate value of that parameter, how probable was the data actually observed? Maximum likelihood estimation simply finds the parameter value that makes this likelihood as large as possible — the value under which the data actually seen was the most plausible outcome, out of every possible parameter value that could have been true. It's a remarkably general principle, applicable to almost any statistical model with unknown parameters, which is part of why it shows up so pervasively across the field.

Worked Example: Estimating a Coin's Bias

Problem: A coin is flipped 10 times, landing heads 7 times. Using maximum likelihood estimation, find the best estimate of p, the coin's true probability of heads.

The probability of observing exactly 7 heads in 10 flips, for any candidate value of p, is given by the binomial distribution formula — this is the likelihood function, viewed as a function of p rather than of the outcome:

To find the p that maximizes this, take the natural log (which turns the product into a sum and doesn't change where the maximum occurs, since log is a strictly increasing function), then take the derivative with respect to p and set it to zero:

d/dp [log L(p)] = 7/p − 3/(1−p) = 0

Solving: 7(1−p) = 3p → 7 − 7p = 3p → 7 = 10p → p̂ = 0.7

The maximum likelihood estimate is exactly 0.7 — precisely the observed proportion of heads (7/10). Checking the log-likelihood at a few nearby candidate values confirms this is genuinely the peak: log L(0.5) ≈ −2.144, log L(0.6) ≈ −1.537, log L(0.7) ≈ −1.321 (the highest, least negative of the four), and log L(0.8) ≈ −1.603 — the likelihood does peak exactly at p=0.7 and falls off on either side, confirming the calculus result numerically.

Why This Matches Intuition

It's a genuinely reassuring feature of MLE that, for this simple case, the method's formal answer (p̂ = 0.7) exactly matches the obvious, intuitive estimate (the raw observed proportion). This isn't a coincidence specific to this example — for many common, simple estimation problems (a proportion, a normal distribution's mean), the maximum likelihood estimate turns out to be exactly the familiar, intuitive formula already used elsewhere on this site. MLE's real value shows up in more complex models — like logistic regression or Poisson regression — where there's no simple, obvious formula to fall back on, and the same underlying "find the parameter that makes the data most probable" principle still applies, just requiring more involved calculus (or, in practice, numerical optimization run by software) to solve.

Where MLE Shows Up Across Statistics

Logistic and Poisson regression: as mentioned above, both fit their coefficients by finding the values that maximize the likelihood of the observed binary or count outcomes, since no simple closed-form least-squares solution exists for these models the way it does for ordinary linear regression.

Estimating distribution parameters generally: fitting any distribution's parameters to observed data — a normal distribution's mean and variance, an exponential distribution's rate, a Poisson distribution's λ — can be approached via MLE, and for many standard distributions, the resulting formula matches the familiar sample statistic already used to estimate that parameter.

The likelihood ratio test: comparing two nested statistical models' relative fit, a common tool in more advanced hypothesis testing, is built directly on comparing their maximized likelihoods against each other.

Real-World Uses

Fitting any statistical model with software: whenever a piece of statistical software reports fitted coefficients for a model without a simple closed-form solution — logistic regression, Poisson regression, and many more advanced models beyond what's covered on this site — MLE (or a close numerical relative of it) is very likely the estimation method working behind the scenes, even when the software output never says so explicitly.

Survival analysis: estimating parameters that describe how long something lasts before an event occurs (a component's lifetime, a patient's time to recovery) commonly relies on MLE, particularly because survival data often includes "censored" observations (subjects who hadn't yet experienced the event when the study ended), which MLE handles naturally by incorporating the right likelihood contribution for each type of observation.

Machine learning model training: many machine learning models, including the logistic regression classifier discussed in statistics in machine learning, are trained by directly maximizing a likelihood function (or an equivalent, minimizing its negative) — the same core principle covered here, just applied at a much larger computational scale.

Common Mistakes

Confusing likelihood with probability. Probability asks "given a fixed parameter, how likely is this data?" Likelihood flips the question: "given this observed, fixed data, how likely is each candidate parameter value?" The math looks similar, but the roles of what's fixed and what's varying are reversed — a subtle but important conceptual distinction.

Assuming MLE always has a clean, closed-form solution. The coin-flip example above solves neatly with basic calculus, but most real MLE problems (like logistic regression's coefficients) have no simple algebraic solution and require iterative numerical methods, run by software, to find the maximizing values.