Ridge and Lasso Regression: Shrinking Coefficients on Purpose

Multiple linear regression mentions ridge regression as a fix for multicollinearity without explaining how it actually works. Ridge and lasso regression both modify ordinary least squares by adding a penalty for large coefficients, deliberately shrinking them toward zero — a small sacrifice in how well the model fits its own training data, in exchange for a model that's more stable and often generalizes better to new data.

The Core Idea: A Penalty for Large Coefficients

Ordinary least squares chooses coefficients that minimize the sum of squared errors (SSE) alone. Ridge regression instead minimizes SSE plus a penalty proportional to the sum of the squared coefficients:

The λ (lambda) parameter controls how strongly large coefficients are penalized. At λ=0, ridge regression reduces exactly to ordinary least squares; as λ grows, coefficients are pushed progressively closer to zero, trading a worse fit on the training data for a model less sensitive to the specific quirks of that data.

Worked Example

Problem: Using the ice cream shop data from simple linear regression — temperature predicting sales, with Sxx=91.2 and Sxy=372.0 — compare the ordinary least squares slope to the ridge regression slope at several values of λ. (For a single predictor with no intercept penalty, the ridge solution simplifies to β = Sxy/(Sxx+λ).)

As λ increases from 0 to 200, the slope shrinks steadily from the ordinary least squares value of 4.079 down to 1.277 — less than a third of its original size. This is ridge regression's defining behavior made concrete: the penalty term directly trades away some of the coefficient's magnitude (and therefore some fit to the training data) in exchange for a smaller, more conservative estimate, less likely to be overly influenced by noise specific to this particular sample.

Why Shrinking Coefficients Can Actually Help

This might seem like a strange thing to want — deliberately making the model fit its own training data worse. The value shows up specifically when predictors are highly correlated with each other (multicollinearity, covered in multiple linear regression) or when there are many predictors relative to the amount of data available. In these situations, ordinary least squares coefficients can become wildly unstable — small changes in the training data can swing them dramatically, even flipping their sign — a symptom of the model overfitting to noise. Ridge regression's penalty constrains this instability directly, typically producing coefficients that, while individually biased toward zero, generalize more reliably to new data than the unconstrained, unstable ordinary least squares estimates would. This bias-for-stability tradeoff is exactly the same bias-variance logic that shows up throughout statistics in machine learning, just applied directly to the coefficients themselves rather than to overall model complexity.

Lasso: The Key Difference From Ridge

Lasso regression uses a similar idea with one crucial change — it penalizes the sum of the absolute values of the coefficients rather than their squares:

This seemingly small change in the penalty (absolute value instead of square) has a significant practical consequence: lasso regression can shrink some coefficients all the way to exactly zero, effectively removing those predictors from the model entirely, while ridge regression's penalty shrinks coefficients toward zero but essentially never sets them to exactly zero. This makes lasso useful as a form of automatic feature selection — with enough predictors and a strong enough penalty, lasso will identify a smaller subset it considers genuinely useful, while ridge regression keeps every predictor in the model, just with smaller, more conservative coefficients across the board.

Choosing λ

λ is not something derived from a formula — it's chosen using k-fold cross-validation, trying a range of candidate λ values and selecting whichever one produces the best average performance on held-out folds. Too small a λ barely differs from ordinary least squares and doesn't address instability; too large a λ shrinks coefficients so much the model underfits, missing genuine signal in the predictors. Cross-validation gives a principled, data-driven way to find a reasonable middle ground rather than guessing.

Real-World Uses

Genomics and high-dimensional data: studies with thousands of genetic predictors but comparatively few subjects are a classic setting where ordinary least squares becomes unreliable — ridge and lasso regression are standard tools in this field precisely because they remain usable even when the number of predictors approaches or exceeds the sample size.

Finance and economics: models built on many correlated economic indicators (interest rates, inflation measures, market indices) routinely suffer from multicollinearity — ridge regression is a standard way to stabilize such models without discarding potentially useful, if correlated, predictors.

Feature selection in predictive modeling: when a dataset has many candidate predictors and it's unclear which genuinely matter, lasso's tendency to zero out weak predictors is routinely used as a data-driven feature-selection step before building a final, simpler model.

Common Mistakes

Choosing λ by looking at which value gives the best fit on the training data alone. The training-data fit always improves as λ approaches 0 (since that approaches unconstrained ordinary least squares) — λ must be chosen using cross-validated performance on data the model wasn't fit to, not raw training fit.

Applying ridge or lasso without standardizing predictors first. Because the penalty is applied directly to the coefficients' size, predictors measured on very different scales (say, age in years versus income in dollars) would be penalized unfairly relative to each other unless first standardized to a comparable scale — this preprocessing step is essential, not optional, for either method.