# PCA is just an SVD

By @omar · Published Nov 2, 2026 · From the roadmap [Machine Learning from First Principles](https://ornix.io/u/omar/machine-learning-first-principles)

Principal component analysis is sold as a way to "find the directions of greatest variance". That is true, and it hides the fact that you already own the tool that computes them. The principal components of a centred data matrix are its right singular vectors. Once you see this, PCA stops being a recipe and becomes a corollary of week 1.

## What PCA asks for

You have $n$ points in $d$ dimensions, stacked as rows of $X \in \mathbb{R}^{n \times d}$ with every column centred to mean zero. Find a unit vector $v_1$ so that projecting the data onto it keeps as much variance as possible:

$$v_1 = \arg\max_{\lVert v \rVert = 1} \frac{1}{n-1}\lVert X v \rVert^2 = \arg\max_{\lVert v \rVert = 1} v^{\top} C v, \qquad C = \frac{X^{\top}X}{n-1}$$

$C$ is the covariance matrix. The maximizer of $v^{\top}Cv$ over unit vectors is the eigenvector of $C$ with the largest eigenvalue, and the maximum value is that eigenvalue. The second component is the best unit vector orthogonal to the first, and so on.

## The SVD connection

Take the singular value decomposition of the data matrix itself:

$$X = U \Sigma V^{\top}$$

where $U$ and $V$ have orthonormal columns and $\Sigma$ is diagonal with $\sigma_1 \ge \sigma_2 \ge \dots \ge 0$. Substituting into the covariance:

$$C = \frac{X^{\top}X}{n-1} = \frac{V \Sigma U^{\top} U \Sigma V^{\top}}{n-1} = V \,\frac{\Sigma^2}{n-1}\, V^{\top}$$

That is an eigendecomposition of $C$ written out. So:

- The principal directions are the columns of $V$ (the right singular vectors).
- The variance along component $k$ is $\sigma_k^2 / (n-1)$.
- The coordinates of the points in the new basis, the scores, are $XV = U\Sigma$.

```mermaid
%% ornix:id=mlk1d006
flowchart TD
    X["Data X: n by d"] --> Cen["Centre each column"]
    Cen --> A["Route A: covariance C, then eigendecomposition"]
    Cen --> B["Route B: SVD of centred X directly"]
    A --> V["Directions V and variances lambda"]
    B --> V
    V --> S["Scores = X times V"]
    S --> R["Keep first k columns: compressed data"]
```

## Five points, two routes

Take $X$ with rows $(2,1), (3,3), (4,3), (6,5), (7,6)$. The column means are $(4.4, 3.6)$. After centring, the covariance matrix is

$$C = \begin{pmatrix} 4.3 & 3.95 \\ 3.95 & 3.8 \end{pmatrix}$$

**Route A.** The eigenvalues of $C$ are $8.008$ and $0.092$. The first eigenvector is $(0.729, 0.684)$ and the second is $(-0.684, 0.729)$ (up to sign).

**Route B.** The SVD of the centred matrix has singular values $5.660$ and $0.607$. Square them and divide by $n - 1 = 4$: $5.660^2/4 = 8.008$ and $0.607^2/4 = 0.092$. Identical to route A. The rows of $V^{\top}$ are the same two vectors.

The first component explains $8.008 / (8.008 + 0.092) = 98.9\%$ of the variance. The five points lie almost on the line through the mean in direction $(0.729, 0.684)$, and their scores along it are about $-3.53, -1.43, -0.70, +2.12, +3.54$. Dropping the second component loses a squared reconstruction error of $\sigma_2^2 = 0.607^2 \approx 0.368$ in total. In general, the squared error from keeping $k$ components is the sum of the discarded $\sigma_j^2$, and no other rank-$k$ linear approximation does better (the Eckart-Young theorem).

```python
import numpy as np

X = np.array([[2, 1], [3, 3], [4, 3], [6, 5], [7, 6]], dtype=float)
Xc = X - X.mean(axis=0)
n = len(X)

C = Xc.T @ Xc / (n - 1)
eigvals, eigvecs = np.linalg.eigh(C)
eigvals, eigvecs = eigvals[::-1], eigvecs[:, ::-1]

U, S, Vt = np.linalg.svd(Xc, full_matrices=False)

print(eigvals)                  # [8.008 0.092]
print(S ** 2 / (n - 1))         # [8.008 0.092]
print(np.abs(eigvecs) - np.abs(Vt.T))  # ~0: same directions up to sign
scores = Xc @ Vt.T
print(scores[:, 0])             # coordinates along the first component
```

## Why use the SVD in practice

Forming $X^{\top}X$ squares the condition number, which loses precision. The SVD works on $X$ directly, and truncated or randomized versions give you only the top $k$ components without computing all of them. That is what scikit-learn's `PCA` does.

## Common mistakes

- **Forgetting to centre.** Without centring, the first singular vector mostly points at the mean instead of at the direction of greatest spread.
- **Mixing units.** If one feature is in metres and another in millimetres, the large-scale feature dominates the variance. Standardize first unless the units are comparable by design.
- **Fitting PCA on all the data before splitting.** The components then carry information from the validation set. Put PCA inside the pipeline and fit it on training folds only.
- **Reading components as causes.** A component is a linear mix of features that varies together. The sign of a component is arbitrary; flip it and nothing changes.
- **Assuming high variance means high signal.** PCA is unsupervised. The label may live in a low-variance direction that you just threw away.

## Check yourself

- Show that $X^{\top}X = V\Sigma^2 V^{\top}$ and state which matrix holds the principal directions.
- If the singular values are 10, 5, 1, what fraction of the variance do the first two components keep?
- Why is the sign of a principal component not meaningful?
- Why does PCA before a train/validation split leak?

---

Roadmap: https://ornix.io/u/omar/machine-learning-first-principles (Markdown: https://ornix.io/u/omar/machine-learning-first-principles.md)

Canonical page: https://ornix.io/u/omar/machine-learning-first-principles/pca-is-just-an-svd
