PCA is just an SVD
Principal component analysis is sold as a way to "find the directions of greatest variance". That is true, and it hides the fact that you already own the tool that computes them. The principal components of a centred data matrix are its right singular vectors. Once you see this, PCA stops being a recipe and becomes a corollary of week 1.
What PCA asks for
You have $n$ points in $d$ dimensions, stacked as rows of $X \in \mathbb{R}^{n \times d}$ with every column centred to mean zero. Find a unit vector $v_1$ so that projecting the data onto it keeps as much variance as possible:
$$v_1 = \arg\max_{\lVert v \rVert = 1} \frac{1}{n-1}\lVert X v \rVert^2 = \arg\max_{\lVert v \rVert = 1} v^{\top} C v, \qquad C = \frac{X^{\top}X}{n-1}$$
$C$ is the covariance matrix. The maximizer of $v^{\top}Cv$ over unit vectors is the eigenvector of $C$ with the largest eigenvalue, and the maximum value is that eigenvalue. The second component is the best unit vector orthogonal to the first, and so on.
The SVD connection
Take the singular value decomposition of the data matrix itself:
$$X = U \Sigma V^{\top}$$
where $U$ and $V$ have orthonormal columns and $\Sigma$ is diagonal with $\sigma_1 \ge \sigma_2 \ge \dots \ge 0$. Substituting into the covariance:
$$C = \frac{X^{\top}X}{n-1} = \frac{V \Sigma U^{\top} U \Sigma V^{\top}}{n-1} = V ,\frac{\Sigma^2}{n-1}, V^{\top}$$
That is an eigendecomposition of $C$ written out. So:
- The principal directions are the columns of $V$ (the right singular vectors).
- The variance along component $k$ is $\sigma_k^2 / (n-1)$.
- The coordinates of the points in the new basis, the scores, are $XV = U\Sigma$.
Five points, two routes
Take $X$ with rows $(2,1), (3,3), (4,3), (6,5), (7,6)$. The column means are $(4.4, 3.6)$. After centring, the covariance matrix is
$$C = \begin{pmatrix} 4.3 & 3.95 \ 3.95 & 3.8 \end{pmatrix}$$
Route A. The eigenvalues of $C$ are $8.008$ and $0.092$. The first eigenvector is $(0.729, 0.684)$ and the second is $(-0.684, 0.729)$ (up to sign).
Route B. The SVD of the centred matrix has singular values $5.660$ and $0.607$. Square them and divide by $n - 1 = 4$: $5.660^2/4 = 8.008$ and $0.607^2/4 = 0.092$. Identical to route A. The rows of $V^{\top}$ are the same two vectors.
The first component explains $8.008 / (8.008 + 0.092) = 98.9%$ of the variance. The five points lie almost on the line through the mean in direction $(0.729, 0.684)$, and their scores along it are about $-3.53, -1.43, -0.70, +2.12, +3.54$. Dropping the second component loses a squared reconstruction error of $\sigma_2^2 = 0.607^2 \approx 0.368$ in total. In general, the squared error from keeping $k$ components is the sum of the discarded $\sigma_j^2$, and no other rank-$k$ linear approximation does better (the Eckart-Young theorem).
import numpy as np
X = np.array([[2, 1], [3, 3], [4, 3], [6, 5], [7, 6]], dtype=float)
Xc = X - X.mean(axis=0)
n = len(X)
C = Xc.T @ Xc / (n - 1)
eigvals, eigvecs = np.linalg.eigh(C)
eigvals, eigvecs = eigvals[::-1], eigvecs[:, ::-1]
U, S, Vt = np.linalg.svd(Xc, full_matrices=False)
print(eigvals) # [8.008 0.092]
print(S ** 2 / (n - 1)) # [8.008 0.092]
print(np.abs(eigvecs) - np.abs(Vt.T)) # ~0: same directions up to sign
scores = Xc @ Vt.T
print(scores[:, 0]) # coordinates along the first component
Why use the SVD in practice
Forming $X^{\top}X$ squares the condition number, which loses precision. The SVD works on $X$ directly, and truncated or randomized versions give you only the top $k$ components without computing all of them. That is what scikit-learn's PCA does.
Common mistakes
- Forgetting to centre. Without centring, the first singular vector mostly points at the mean instead of at the direction of greatest spread.
- Mixing units. If one feature is in metres and another in millimetres, the large-scale feature dominates the variance. Standardize first unless the units are comparable by design.
- Fitting PCA on all the data before splitting. The components then carry information from the validation set. Put PCA inside the pipeline and fit it on training folds only.
- Reading components as causes. A component is a linear mix of features that varies together. The sign of a component is arbitrary; flip it and nothing changes.
- Assuming high variance means high signal. PCA is unsupervised. The label may live in a low-variance direction that you just threw away.
Check yourself
- Show that $X^{\top}X = V\Sigma^2 V^{\top}$ and state which matrix holds the principal directions.
- If the singular values are 10, 5, 1, what fraction of the variance do the first two components keep?
- Why is the sign of a principal component not meaningful?
- Why does PCA before a train/validation split leak?