Principal Component Analysis (PCA): A Step-by-Step Two-Variable Example

Datasets often carry more columns than they carry information. Height and weight, arm length and leg length, or a dozen survey questions about the same attitude all move together, so much of what they record is repetition. Principal component analysis (PCA) finds new, uncorrelated variables, called principal components, that are weighted combinations of the originals and are ordered so that the first captures as much of the total variability as possible, the second captures as much of what remains, and so on. Keeping only the first few components compresses the data with little loss, and plotting them often reveals structure that a table of raw columns hides.

The Idea

Picture a scatterplot of two positively related variables: an elongated cloud tilted along the diagonal. The first principal component is the direction of that long axis, the direction in which the points are most spread out. The second component is perpendicular to it, capturing the narrower spread across the cloud. PCA is a rotation of the axes to line up with the cloud's natural directions, and each observation gets new coordinates, its scores, along the rotated axes.

Algebraically, the components come from the eigenvectors of the covariance matrix (see covariance), and the variance along each component equals the matching eigenvalue. Because rotating axes does not change how far points are from each other, the total variance is preserved: the eigenvalues add up to the total variance of the original variables.

Worked Example

Problem: Six people are measured on two tests, X and Y. Run a PCA.

Person	1	2	3	4	5	6
X	2	3	5	6	8	10
Y	3	5	4	8	7	11

Step 1 — Standardize. The means are x̄ = 5.667 and ȳ = 6.333, and the standard deviations are 3.011 and 2.944. Converting each value to a z-score puts both variables on the same footing. For person 1, zₓ = (2 − 5.667) ÷ 3.011 = −1.218 and z_y = (3 − 6.333) ÷ 2.944 = −1.132.

Step 2 — The correlation matrix. The covariance matrix of standardized variables is the correlation matrix. Here the correlation between X and Y is r = 0.895, so:

Step 3 — Eigenvalues and eigenvectors. For a two-variable correlation matrix, solving (1 − λ)² − r² = 0 gives the eigenvalues λ = 1 + r and λ = 1 − r, so λ₁ = 1.895 and λ₂ = 0.105. Their eigenvectors are always the same for two standardized variables: (1, 1) ÷ √2 ≈ (0.707, 0.707) for the first component and (−1, 1) ÷ √2 for the second.

Step 4 — Variance explained. The eigenvalues add to 2, the total variance of two standardized variables. The first component explains 1.895 ÷ 2 = 94.7% of it, and the second only 5.3%. Nearly all the information in the two tests can be summarized by one number per person.

Step 5 — Component scores. Each person's scores are weighted sums of their z-scores: PC1 = (zₓ + z_y) ÷ √2 and PC2 = (z_y − zₓ) ÷ √2.

Person	zₓ	zy	PC1	PC2
1	−1.218	−1.132	−1.662	0.060
2	−0.886	−0.453	−0.946	0.306
3	−0.221	−0.793	−0.717	−0.404
4	0.111	0.566	0.479	0.322
5	0.775	0.226	0.708	−0.388
6	1.439	1.585	2.139	0.103

As a check, the variance of the PC1 column is 1.895 and of the PC2 column is 0.105, exactly the eigenvalues, and the two columns are uncorrelated by construction. Their variances sum to 2, unchanged from the original.

Interpreting the Components

The loadings, the weights on each variable, tell you what a component means. PC1 weights both tests equally and positively, so it is an "overall level" score: high for people strong on both tests (person 6: 2.14), low for people weak on both (person 1: −1.66). PC2 contrasts the two tests, with a positive score meaning the person did relatively better on Y than on X, and it is small for everyone, reflecting that the tests mostly agree. Discarding PC2 and keeping only PC1 would replace the two columns with one, losing just 5.3% of the variability. This is the sense in which PCA compresses data.

Why Scale Matters

PCA seeks directions of maximum variance, so variables with large numeric spread dominate unless they are standardized. Try running the analysis on the raw covariance matrix instead. With the original data, X and Y have similar spreads, and the results are almost the same (PC1 explains 94.8%, with loadings of about 0.72 on X and 0.70 on Y). Now suppose Y had been recorded on a scale ten times larger, as when one variable is in dollars and another in thousands of dollars. The covariance matrix becomes [[9.07, 79.3], [79.3, 866.7]]. Its eigenvalues are 873.9 and 1.79, so PC1 "explains" 99.8% of the variance, but its loading is about (0.09, 0.996): the component is essentially just Y. The variance explained reflects units, not structure. Standardizing first, equivalent to working from the correlation matrix, removes the problem, and it is the default recommendation when variables are in different units.

How Many Components to Keep

With p variables there are p components, and the goal is to keep the few that matter. Common guides are:

• Cumulative variance explained: keep enough components to reach a target such as 80% or 90%.
• Scree plot: plot the eigenvalues in order and look for the "elbow," where the curve flattens.
• Kaiser criterion: for standardized data, keep components with eigenvalue above 1, meaning they carry more variance than a single original variable. Here that keeps only PC1 (1.895).

None of these is exact. The choice depends on the purpose, and cross-validation of downstream performance is often a good tie-breaker, as described in k-fold cross-validation.

Limitations

• It is linear. PCA finds straight-line directions and misses curved structure.
• It is sensitive to outliers. A few extreme points can pull a component toward themselves, so check the data first, as in outlier detection.
• Components can be hard to interpret. Each is a mixture of all the variables, and large loadings are the only guide to meaning.
• Variance is not the same as importance. A component with small variance might still be the one that predicts your outcome.

Real-World Uses

Exploratory analysis: plotting the first two components of a many-variable dataset often reveals clusters or outliers, an extension of exploratory data analysis.

Regression with correlated predictors: replacing highly correlated predictors with a few uncorrelated components sidesteps the multicollinearity problem in multiple regression, at the cost of coefficients that are harder to interpret.

Machine learning: dimension reduction before modelling, noise reduction, and data compression, all discussed in statistics in machine learning.

Common Mistakes

Skipping standardization. When variables have different units or scales, the largest-spread variable takes over the first component.

Reading a component as a cause. PCA summarizes how variables co-vary. It does not identify latent causes, which is the aim of factor analysis, a related but distinct technique.

Applying it to uncorrelated variables. If the variables are already nearly uncorrelated, every component carries similar variance and nothing can be compressed.

Forgetting the scores' meaning is relative. Component scores are centred at zero, so a score of 0 means average, not "none" of anything.

What is the difference between PCA and regression?

Regression predicts one variable from others and minimizes vertical errors. PCA has no outcome variable: it finds the directions of greatest spread and minimizes perpendicular distances to the fitted line or plane.

Are eigenvalues and variance the same thing?

For principal components, yes: each eigenvalue equals the variance of the scores on its component, so eigenvalue divided by their total gives the share of variance explained.

Should I use the covariance or correlation matrix?

Use the correlation matrix, that is, standardize, when variables are measured in different units or have very different spreads. The covariance matrix is appropriate when all variables share the same units and their raw spread is meaningful.

Does PCA need normally distributed data?

No, PCA itself only uses variances and covariances. Normality matters only if you want to attach probability statements or tests to the results.