Feature scaling

Setup#

pip install scikit-learn numpy

Outputs below verified with scikit-learn 1.7.2 and NumPy 1.26.

Watch a model ignore a column#

The label here depends on age — the older customers bought health insurance. Income is noise. Watch what an unscaled distance model does with that.

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

# monthly income in rupees, age in years -- two wildly different scales
X = np.array([[92000, 24], [23000, 52], [88000, 25], [21000, 49],
              [95000, 26], [24000, 51], [90000, 27], [25000, 50]])
y = np.array([0, 1, 0, 1, 0, 1, 0, 1])   # 1 = bought health insurance (the older half)

new_customer = np.array([[91000, 53]])   # high income AND older

knn = KNeighborsClassifier(n_neighbors=3).fit(X, y)
print("unscaled prediction:", knn.predict(new_customer)[0])

scaler = StandardScaler().fit(X)   # learns each column's mean and spread
knn2 = KNeighborsClassifier(n_neighbors=3).fit(scaler.transform(X), y)
print("scaled prediction:  ", knn2.predict(scaler.transform(new_customer))[0])

print(f"income: mean {scaler.mean_[0]:.0f}, std {scaler.scale_[0]:.0f}")
print(f"age:    mean {scaler.mean_[1]:.0f}, std {scaler.scale_[1]:.1f}")
print("new customer, scaled:", scaler.transform(new_customer).round(2))

unscaled prediction: 0
scaled prediction:   1
income: mean 57250, std 34065
age:    mean 38, std 12.5
new customer, scaled: [[0.99 1.2 ]]

The walkthrough#

The unscaled model got it wrong. The new customer is 53 — squarely in the group that buys insurance. But the model answered 0. Look at the printed spreads to see why. Income varies by around 34,065; age varies by around 12.5. In a distance calculation, an income gap of 3,000 rupees outweighs an age gap of 25 years. The three nearest neighbours were the three high-income young people. Age never got a vote.

StandardScaler learns, then applies. fit records each column's mean and spread. transform rewrites each value as "how many spreads away from the mean". After that, the new customer reads as [0.99, 1.2] — a bit rich, notably old — and both facts carry similar weight.

Fit on training data only. The scaler is part of the model. If you compute the mean using test rows, information about the test set leaks into training. The pattern is always: fit on train, transform on both.

Other scalers, one line each#

Common mistakes#

Calling fit_transform on the test set. This re-learns the mean and spread from test data. Results look fine and are quietly wrong. Use transform alone on anything the model will be judged on. See train-test split for why this separation is sacred.

Scaling with outliers using MinMaxScaler. One billionaire in the income column squashes everyone else into a tiny sliver near 0. RobustScaler uses the median and the middle 50% spread, so one absurd row cannot distort the rest.

Scaling columns that are already 0 or 1. One-hot columns do not need it, and scaling them can make sparse data dense and huge. Scale the numeric columns, pass the rest through — ColumnTransformer exists for exactly this.

Scaling before a tree model and expecting a change. Decision trees, random forests and XGBoost split on thresholds. "income > 50,000" and "scaled_income > -0.2" are the same split. Scaling neither helps nor hurts them.

Try it yourself#

Change the new customer to [[24000, 25]] — poor and young. Predict what both models will say before running. Then break the scaled model on purpose: multiply the income column by 1,000 after scaling and watch age get ignored again.

What to learn next#

• Power transforms — when a column's problem is its shape, not its size.
• Feature engineering — building the columns worth scaling.
• Train-test split — why the scaler must never see the test set.