Principal Component Analysis: Finding the Best Few Dimensions in a Cloud of Data

Take a dataset with many measured quantities — a thousand genes per person, every pixel of an image, every word frequency in a document. The dimensionality is large; the structure is hopelessly tangled. But often, only a handful of dimensions actually matter. The data lives, approximately, on a low-dimensional subspace of the full space, and most of the apparent complexity is irrelevant noise.

Principal Component Analysis (PCA) is the standard method for finding that low-dimensional subspace. It identifies the directions in the data space along which the variance is largest, ranks them, and lets you project the data onto the top few — collapsing thousands of dimensions to two, three, or twenty without losing the structural information that distinguishes one data point from another.

Karl Pearson introduced PCA in 1901; Harold Hotelling rediscovered it in 1933 under the modern name. A century later, it is one of the most-used algorithms in all of applied mathematics: every statistical software package implements it, nearly every data analysis pipeline starts with it, and a wide range of scientific fields — genomics, finance, image processing, neuroscience, climate science — would be unimaginably harder without it. This article is about what PCA does, why it works, and where its uses have spread.

The geometric picture

Suppose you have data points scattered in a two-dimensional plane. The points form a cloud — not uniform, not random, but stretched in some direction. The picture below shows $60$ such points scattered around the origin, with a clear elongated structure running diagonally.

The blue line is the first principal component (PC1) — the direction in which the data has the most spread. The red line, perpendicular to it, is the second principal component (PC2) — the direction with the most remaining variance after PC1 is removed. The lengths of the two lines are proportional to the square roots of the eigenvalues of the covariance matrix — equivalently, to the standard deviations of the data along each axis. In this example, PC1 captures about $93\%$ of the total variance and PC2 captures the remaining $7\%$, so projecting onto PC1 alone preserves almost all the structure of the data.

In higher dimensions the same idea applies. The principal components are mutually orthogonal directions, ordered by how much variance they capture. Project the data onto the top $k$ of them and you obtain a $k$-dimensional summary that loses as little variance as possible.

The math

Let $X$ be an $n \times d$ data matrix: $n$ samples, each a vector in $d$-dimensional space. Center the data by subtracting the mean of each column. The covariance matrix of the centred data is the $d \times d$ matrix

$\Sigma \;=\; \tfrac{1}{n}\,X^T X.$

The diagonal of $\Sigma$ holds the variances of the individual coordinates; the off-diagonal entries hold the covariances between pairs of coordinates. PCA performs the spectral decomposition of $\Sigma$:

$\Sigma \;=\; V \Lambda V^T,$

where $V$ is an orthogonal matrix whose columns are the eigenvectors of $\Sigma$ (the principal components), and $\Lambda$ is a diagonal matrix of eigenvalues. The eigenvalues, conventionally written in decreasing order $\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_d \geq 0$, are the variances of the projected data along each principal component.

To project the data onto the top $k$ principal components: form $V_k$, the matrix whose columns are the first $k$ columns of $V$, and compute $Y = X V_k$. The rows of $Y$ are the $k$-dimensional projections of the data points. The fraction of total variance preserved is $\sum_{i \leq k} \lambda_i / \sum_i \lambda_i$.

Why variance maximisation works

The choice to maximise variance — rather than some other criterion — is motivated by a clean optimisation result. Among all unit vectors $\mathbf{v}$, the one that maximises the variance of the projected data $X\mathbf{v}$ is exactly the eigenvector of $\Sigma$ corresponding to the largest eigenvalue. The proof is a one-line application of the Rayleigh quotient: the quadratic form $\mathbf{v}^T \Sigma \mathbf{v}$ is maximised over $\|\mathbf{v}\| = 1$ at the top eigenvector, with value equal to the top eigenvalue.

Iterating: among unit vectors orthogonal to the first principal component, the variance is maximised by the second eigenvector. Continue until all $d$ eigenvectors are extracted. The principal components are therefore the unique orthogonal basis of $\mathbb{R}^d$ that successively maximises variance.

This explanation also clarifies the connection between PCA and the singular value decomposition (SVD). The SVD of the centred data matrix is $X = U \Sigma_X V^T$, where $V$ contains the principal components, $\Sigma_X$ is a diagonal matrix of singular values, and the squares of the singular values are the eigenvalues of $X^T X = n \Sigma$. In modern implementations PCA is almost always computed via SVD, because SVD is more numerically stable than forming $X^T X$ explicitly.

Applications

The breadth of PCA’s applications is hard to overstate.

Genomics. A single sequencing experiment can produce expression levels for $20{,}000$ genes per person. PCA projects this $20{,}000$-dimensional data into two or three dimensions where it can be plotted. Patterns of ethnicity, disease status, or population structure routinely emerge in the top few principal components — Cavalli-Sforza’s pioneering 1990s work showed that PCA on human genetic data recovers the main features of human migration patterns from genetic similarity alone.

Image compression and recognition. PCA on collections of images extracts the dominant patterns of variation. The 1991 paper of Turk and Pentland introduced eigenfaces: PCA on a database of facial photographs yielded a small set of facial templates that, in combination, reconstructed all faces in the database. New faces could be recognised by their projections onto this small basis. Eigenfaces dominated facial recognition for nearly a decade, until deep learning overtook them.

Finance. Stock returns are correlated, and the dominant pattern of correlation is the overall market direction — the “market factor”. PCA on stock-return matrices recovers this factor (the first principal component is typically a roughly uniform weighting of all stocks) and identifies smaller factors like sector trends and style trends. Modern quantitative finance still uses PCA-derived risk factors.

Climate science. Sea-surface temperature data is high-dimensional (one measurement per grid cell). PCA reveals the dominant climate modes: the leading principal component of Pacific sea-surface temperature is El Niño, the second is the Pacific Decadal Oscillation, and so on. These modes were identified by climate scientists exactly through eigenvalue decompositions of multi-decadal datasets.

Neuroscience. Brain-imaging datasets (fMRI, EEG) have very high dimensionality. PCA reduces them to interpretable summary axes, often revealing neural activity patterns that correspond to identifiable cognitive states.

Visualisation. Beyond specific applications, PCA is the default tool for “what does my dataset look like?” Almost every data scientist’s first step in exploring a new dataset is to project it onto the top two principal components and plot the result, before trying any more sophisticated analysis.

What PCA cannot do

PCA is a linear method and inherits the limitations of linearity. The principal components are straight lines (or hyperplanes) through the data; if the meaningful structure of the data is curved or nonlinear, PCA misses it. The classic example is the “Swiss roll” dataset: data sampled from a sheet of paper rolled into a spiral. PCA on this data finds the axes of the bounding box but not the underlying two-dimensional structure of the sheet.

Several extensions address this. Kernel PCA applies PCA in a high-dimensional feature space implicitly defined by a kernel function, enabling capture of nonlinear patterns. Manifold learning methods like t-SNE and UMAP try to preserve local distances or topological structure rather than global variance. Autoencoders — neural networks trained to compress and reconstruct data — give a nonlinear analogue of PCA that has dominated dimensionality reduction in the deep-learning era.

PCA also relies on the data being mean-centred, and on the features having comparable scales. PCA on raw features where one feature has variance a thousand times larger than the others will simply recover that feature as the top component. Standardising the data (mean $0$, standard deviation $1$ per feature) before PCA is essential in nearly every practical case.

A century-old method that has not aged

PCA is, by any honest accounting, one of the most-used mathematical techniques in the world. Every modern programming language has a standard PCA implementation. Every statistics textbook teaches it. Every data analyst uses it at some point in nearly every analysis. The original Pearson 1901 paper is, more than a century later, one of the most-cited statistical papers ever written.

The reason for the longevity is the simplicity-to-power ratio. The mathematics is no more than spectral decomposition of a symmetric positive-semidefinite matrix — an undergraduate linear algebra calculation. The result is a complete decomposition of the data into orthogonal directions ranked by importance, with the importance precisely quantified by the eigenvalues. There is no parameter to tune (other than how many components to keep), no model to fit, no convergence to worry about. Just an eigenvalue decomposition and a projection.

That so much practical value can flow from such elementary mathematics is a recurring lesson of applied mathematics. The PCA story is one of the cleanest examples. A century and a quarter ago, Karl Pearson asked a clean question — what is the best low-dimensional fit to a cloud of data? — and gave a clean answer using the eigenvectors of a covariance matrix. Every data scientist working today still uses essentially that answer, every day. Not bad for one short 1901 paper.

Frequently asked

Who invented PCA?

Karl Pearson, the founding statistician, introduced the basic idea in 1901 in a paper titled 'On lines and planes of closest fit to systems of points in space'. He framed it as the problem of finding the line, plane, or hyperplane that best fits a cloud of data points. Harold Hotelling rediscovered and formalised the technique in 1933 under the modern name 'principal components', framing it as a way to summarise multivariate data. The mathematical content — eigenvectors and eigenvalues of the covariance matrix — was the same in both treatments, but Hotelling's framing made it a standard tool of psychology and the social sciences. The two papers are often cited together as the foundational works.

What is the relationship between PCA and the singular value decomposition (SVD)?

They are essentially the same calculation done with different language. The SVD of the centred data matrix X is X = U Σ V^T, where the columns of V are the principal components, the singular values in Σ are the square roots of the eigenvalues of the covariance matrix, and the columns of UΣ are the data projected onto the principal components. In modern implementations, PCA is almost always computed via SVD because the SVD is numerically more stable than directly forming the covariance matrix and computing its eigenvalues. The relationship X^T X = V Σ² V^T makes the equivalence precise.

When does PCA fail or mislead?

Several situations. First, when the data is not 'linearly structured' — when the meaningful directions are curved rather than straight, PCA misses them. Kernel PCA and manifold-learning techniques like t-SNE and UMAP address this. Second, when the variance is not the meaningful quantity — for example, when you care about distinguishing categories rather than describing variability, linear discriminant analysis often does better. Third, when the data has features with very different scales: PCA on raw features dominated by one feature with huge variance will simply recover that feature. Standardising features (mean 0, variance 1) before PCA is almost always necessary.

What was the 'eigenfaces' breakthrough?

In 1991, Matthew Turk and Alex Pentland at MIT published a paper showing that PCA applied to a database of facial photographs produces a set of 'eigenfaces' — orthogonal facial templates whose combinations can reconstruct any face in the database with high fidelity. A new face image could be classified by projecting it onto the top few eigenfaces and finding the nearest match. This was the first practical face recognition system to use PCA, and it kicked off a decade of computer vision research based on similar dimensionality-reduction ideas. Modern face recognition uses deep neural networks rather than eigenfaces, but the conceptual breakthrough — that linear projections to a few PCA directions could capture identity — was an important moment in computer vision.

Continue reading

Blog

RSA Encryption: How Two Primes Lock the World's Secrets

Pick two large primes. Multiply them. Publish the product. Every HTTPS connection and every encrypted message rests on the fact that nobody can factor that product back into the two primes.

Read →Blog

Jordan Normal Form: What Matrices Look Like When They Cannot Be Diagonalised

Most matrices can be diagonalised — but not all. For those that resist, Camille Jordan found the next best canonical form: a block-diagonal structure from which much of linear algebra follows.

Read →Blog

Bloom Filters: How to Remember Things You Cannot Afford to Store

Storing a billion entries for membership tests is expensive. A clever probabilistic structure from Burton Bloom in 1970 does it with a few megabytes — at the cost of an occasional false positive.

Read →