The geometry of L²

Center every variable (subtract its mean). Collect all the finite-variance random variables on a space into one set, $L^2(\Omega )$, and equip it with an inner product:

It obeys every inner-product axiom — bilinear, symmetric, with $⟨X,X⟩\ge 0$— so $L^2$ is a Hilbert space and probability turns into geometry. For centered variables, that inner product is exactly the covariance:

Standard deviation is a length; covariance is the inner product between two directions. And the fundamental bound on correlation is nothing but Cauchy–Schwarz:

That one inequality is the entire reason $|\rho |\le 1$.

The angle $\theta$ between two centered variables is defined by

Pearson correlation is the cosine of the angle — and "cosine similarity" on centered data is exactly $\rho$. Now apply the law of cosines, which holds in any inner-product space:

The variance-of-a-difference formula and the law of cosines are the same theorem. Uncorrelated ($\rho =0$) means orthogonal ($\theta =90^{\circ }$), and then variances add like Pythagoras. Independence implies orthogonality; the converse fails, because the angle sees only second moments, never the whole joint law.

Standardization is the one-dimensional move: rescale a variable to unit length.

Its multivariate twin is whitening — rotate and rescale a vector until its covariance is the identity:

That is a change of basis into a frame where the coordinates are uncorrelated and unit-variance. In that frame, plain Euclidean distance becomes Mahalanobis distance back in the original coordinates:

The covariance matrix is the metric. Euclidean distance trusts every direction equally; Mahalanobis trusts the directions the data actually varies in. Cosine distance throws away length entirely, keeping only the angle $\theta$ from section 3.

The covariance matrix $\Sigma =\mathbb{E}[(x-\mu )(x-\mu {)}^{\top }]$ is symmetric and positive semidefinite, so the spectral theorem hands us an orthonormal eigenbasis:

The columns of $Q$ are the principal axes — orthogonal directions of variation — and each ${\lambda }_i$ is the variance along its axis. Whitening from section 4 is simply${\Sigma }^{-1/2}=Q{\Lambda }^{-1/2}{Q}^{\top }$: rotate into the eigenbasis, divide each axis by its standard deviation, rotate back.

PCA is the eigendecomposition of $\Sigma$: keep the eigenvectors with the largest eigenvalues and you keep the directions carrying the most variance. The fraction of variance a component explains is

SVD reaches the same place without ever forming $\Sigma$. For a centered data matrix $X\in {\mathbb{R}}^{n\times p}$ (rows are observations),

The right singular vectors $V$ are the principal axes, and the singular values satisfy${\sigma }_i^2\propto {\lambda }_i$. SVD is the numerically honest way to do PCA, and it extends the spectral theorem to non-square matrices.

Take two independent random vectors in ${\mathbb{R}}^d$. As $d\to \infty$, concentration of measure forces them to be nearly orthogonal:

Distances concentrate as well — nearest and farthest neighbors become almost equidistant, so "closeness" stops carrying information:

This is why dimension reduction is a necessity, not a convenience, and why the eigenbasis of section 5 matters: real data lives near a low-dimensional subspace, and PCA finds it. The spine closes its loop — the cosine that began as correlation is the same cosine that collapses in high dimension.