Kernel PCA

Setup#

pip install scikit-learn

Outputs verified with scikit-learn 1.7.2.

A ring inside a ring#

make_circles creates the classic PCA-killer: one class encircling the other. We score each representation by whether a straight line can separate the classes in it.

from sklearn.datasets import make_circles
from sklearn.decomposition import PCA, KernelPCA
from sklearn.linear_model import LogisticRegression

# a ring inside a ring -- no straight line separates them
X, y = make_circles(n_samples=400, factor=0.3, noise=0.05, random_state=0)

def line_score(Z, y):
    """How well a straight line separates the classes in this space."""
    return LogisticRegression().fit(Z, y).score(Z, y)

print(f"original 2-D space:        {line_score(X, y):.2f}")

Z_pca = PCA(n_components=2).fit_transform(X)
print(f"after ordinary PCA:        {line_score(Z_pca, y):.2f}")

Z_kpca = KernelPCA(n_components=2, kernel="rbf", gamma=4).fit_transform(X)
print(f"after kernel PCA (rbf):    {line_score(Z_kpca, y):.2f}")

original 2-D space:        0.50
after ordinary PCA:        0.50
after kernel PCA (rbf):    1.00

The walkthrough#

0.50 is a coin flip. In the original space, the best straight line gets half the points right — the rings are concentric, so every line cuts both. Ordinary PCA "helps" by rotating a circle, which changes nothing: still 0.50. Kernel PCA's output is perfectly separable: 1.00.

kernel="rbf" is the radial basis function kernel — similarity decays smoothly with distance, like the warmth of a fire fading as you step away. It is the default choice for "I know it curves, I do not know how". Polynomial and cosine kernels suit other shapes.

gamma is the zoom knob, and it is the parameter that decides success. It sets how fast similarity fades with distance. Too small and everything looks similar — the unrolling flattens nothing. Too large and only immediate neighbours look similar — the data shatters into confetti. There is no formula; tune it. A reasonable starting range is 0.1 to 10 on standardised data, searched on a log scale.

No labels were used in the unrolling. line_score uses labels only to measure the spaces. Kernel PCA itself is unsupervised, like PCA — it found the ring structure on its own.

Common mistakes#

Leaving gamma at the default and concluding kernel PCA is useless. The default (1/n_features) is a placeholder, not a recommendation. On this same dataset, gamma=0.01 scores 0.50 — indistinguishable from failure. Sweep it before judging.

Running it on big datasets without thinking. Kernel PCA builds the full similarity table: all pairs of points. 50,000 rows means 2.5 billion entries — memory death long before the maths finishes. Past roughly 10,000 rows, use the Nystroem approximation (sklearn.kernel_approximation.Nystroem) followed by ordinary PCA.

Expecting a clean reverse mapping. PCA can reconstruct originals from components. Kernel PCA's components live in the never-built lifted space, and mapping back (fit_inverse_transform=True) is approximate — a fitted model, not an inversion. Reconstructions can be poor even when the embedding is excellent.

Skipping scaling. The RBF kernel measures distance, and unscaled columns distort distance exactly as they did for k-NN. Standardise first.

Try it yourself#

Sweep gamma through 0.01, 0.1, 1, 4, 20, 100, printing the line score for each. Find the plateau of good values, then explain the failures at both ends using the zoom-knob picture.

What to learn next#

• t-SNE — nonlinear reduction built specifically for visualisation.
• Truncated SVD and LSA — back to linear, but for sparse text matrices.
• Autoencoders — learning the lift instead of choosing a kernel.