Understand why high-dimensional data is fundamentally different. Explore how distances lose meaning, how PCA reveals structure, and why feature scaling matters for ML algorithms.
Space Grows Exponentially
Split each axis into just 10 bins and a d-dimensional grid has 10ᵈ cells to fill, while the ball inscribed in the unit cube shrinks toward zero volume. Data becomes exponentially sparse.
Distances Lose Meaning
As dimensions grow, the ratio of max-to-min pairwise distance approaches 1. All points become “equidistant”: KNN breaks down.
Data Requirements Explode
To maintain the same density, you need exponentially more samples. 100 points in 2D becomes meaningless in 20D: you'd need 10²⁰ samples.
The solution toolkit: Dimensionality reduction (PCA, t-SNE, UMAP), feature selection, regularization, and understanding which algorithms are robust to high dimensions.
Move the slider to increase dimensions from 1 to 100. Watch how distances become meaningless and data grows impossibly sparse.
First 2 coordinates of the 10 sampled points (re-sampled at each dimension)
0.229
Neighbors are close
3.1416
Volume still significant
9.46
Large ratio: distances informative
Principal Component Analysis finds the axes of maximum variance. See how correlated data can be rotated to reveal its true structure.
Original Space (X, Y)
PCA Space (PC1, PC2)
Eigenvalue 1 (PC1)
0.1602
Variance along principal axis
Eigenvalue 2 (PC2)
0.0020
Variance along secondary axis
The correlated structure in the original space is revealed as spread along PC1 in the transformed space. Dropping PC2 retains 98.8% of total variance.
When features have vastly different scales, distance-based algorithms are dominated by the largest-scale feature. Standardization (z-score) fixes this.
Unscaled Data
Scale-biasedStandardized (z-score)
Scale-balancedFormula
z = (x − μ) / σ
When to scale
KNN, SVM, PCA, gradient descent
Scale-invariant
Decision trees, Random Forests
How you measure “distance” fundamentally changes what an algorithm learns. Each metric has assumptions. Choose wisely.
Euclidean Distance
L2d(p,q) = √Σ(pᵢ − qᵢ)²
KNN, K-means, PCA distance matrix. Best when all features are on the same scale.
Manhattan Distance
L1d(p,q) = Σ|pᵢ − qᵢ|
Robust to outliers. Used in LASSO regression, city-block navigation, and high-dim feature spaces.
Cosine Similarity
cos θcos(θ) = (A·B) / (‖A‖·‖B‖)
Text/document similarity, NLP embeddings, recommendation systems. Ignores magnitude.
Mahalanobis Distance
D_MD_M(x) = √((x−μ)ᵀ S⁻¹ (x−μ))
Anomaly detection, accounts for correlations between features. Scale-invariant by design.
Curse of Dimensionality Affects All Distances
In high dimensions, the ratio of max-to-min distance between points approaches 1: all points become nearly equidistant. Euclidean distance suffers most; Manhattan is more robust; Cosine similarity is often preferred for high-dim sparse data like text.
Click any card to see worked examples.
K-means iteratively assigns each point to its nearest centroid, then recalculates centroids. Try different values of K to see how cluster boundaries change.
Inertia
0.1847
Cluster 1
7
Centroid: (0.16, 0.18)
Cluster 2
6
Centroid: (0.50, 0.77)
Cluster 3
7
Centroid: (0.82, 0.46)
Choosing K: The Elbow Method
Lower inertia = tighter clusters. The “elbow” indicates the optimal K. Here K=3 matches the natural structure in this dataset.