Learn / Unsupervised
Dimensionality reduction
Squeezing many features into a few while keeping what matters, for plotting, compression and cleaner inputs.
On this page
Intuition
A hundred features rarely carry a hundred independent pieces of information. Dimensionality reduction finds a smaller set of coordinates that preserves most of the structure. Linear methods (PCA) rotate the data and drop the least informative axes. Non-linear methods (t-SNE, UMAP) try to keep neighbours together, even if that means bending space.
What the data looks like
Each row has many columns. The method keeps the rows and replaces the columns with a few new ones.
| rowcustomer | inputfood | inputfuel | inputtravel | inputonline | input… | outputpc1 | outputpc2 |
|---|---|---|---|---|---|---|---|
| C-3001 | 0.31 | 0.12 | 0.02 | 0.2 | … | 1.84 | -0.22 |
| C-3002 | 0.08 | 0.02 | 0.41 | 0.33 | … | -2.1 | 1.35 |
| C-3003 | 0.29 | 0.15 | 0.01 | 0.18 | … | 1.62 | -0.4 |
| C-3004 | 0.1 | 0.01 | 0.38 | 0.36 | … | -1.95 | 1.21 |
… plus 44 more spending columns.
Fifty columns per row go in; two coordinates per row come out, ready to plot or to feed another model.
Example rows are invented for illustration.
How the data is prepared
- Centre and scale. PCA follows variance, so an unscaled feature in large units becomes the first component by default.
- Remove or cap outliers; a few of them can define a component.
- For t-SNE and UMAP, reduce to 30-50 dimensions with PCA first. It is faster and removes noise.
Algorithms
PCA
Rotate the data onto the directions of greatest variance and keep the first few.
- Cost
- O(n·d²)
- Strength
- Fast, deterministic, invertible.
- Weakness
- Linear: it cannot unroll curved structure.
- Used for
- compression, decorrelating features
Kernel PCA
PCA in an implicit non-linear feature space.
- Cost
- O(n³)
- Strength
- Captures some non-linear structure.
- Weakness
- n×n kernel matrix; kernel choice matters.
- Used for
- non-linear denoising
t-SNE
Place points in 2-D so that neighbours stay neighbours.
- Cost
- O(n²) exact
- Strength
- Excellent local cluster separation.
- Weakness
- Distances between clusters mean little; not reusable on new data.
- Used for
- visualising embeddings
UMAP
Build a neighbour graph and lay it out in low dimensions.
- Cost
- O(n log n)
- Strength
- Fast; keeps more global structure than t-SNE.
- Weakness
- Stochastic; hyper-parameters change the picture.
- Used for
- embedding maps, single-cell data
The maths
PCA finds the unit direction along which the centred data varies most:
The solutions are the eigenvectors of the covariance matrix , and each eigenvalue is the variance captured by that component. The share kept by the first components is .
t-SNE converts distances into neighbour probabilities in the original space and in the map, then moves the map points to minimise the Kullback-Leibler divergence
Perplexity sets roughly how many neighbours each point cares about.
Real-world examples
- Visualising embeddings: seeing whether support tickets or documents form topics.
- Compression and denoising: keeping the top components and discarding the rest.
- Decorrelating features before a model that struggles with collinearity.
Playground
The swiss roll is a flat sheet rolled up in 3-D. PCA can only look at it from one angle, so the layers stay on top of each other. t-SNE and UMAP follow the sheet and unroll it. Drag the 3-D view to see the input.
Drag the 3-D view to rotate it.
Drag the 3-D view to rotate it.
Press Train to compare.
Train a model to see its loss curve.
| Step - | train ━ | test ┅ |
|---|
colour = position along the manifold
Press Train. The verdict appears here.
In Python
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import umap
Xs = StandardScaler().fit_transform(X)
pca = PCA(n_components=0.95).fit(Xs) # keep 95% of the variance
print(pca.n_components_, pca.explained_variance_ratio_[:5])
X50 = PCA(n_components=50).fit_transform(Xs) # denoise first...
X2 = umap.UMAP(n_neighbors=15, min_dist=0.1).fit_transform(X50) # ...then map to 2-D for plotting
Pitfalls
- Reading distances between clusters in t-SNE or UMAP plots. Only local neighbourhoods are meaningful.
- Cluster sizes in those plots are not meaningful either.
- One run, one perplexity. Change the settings and the seed before believing a pattern.
- Using a t-SNE map as model features. It cannot place new points; use PCA or UMAP’s
transform.