Power transforms

Setup#

pip install scikit-learn scipy numpy

Outputs verified with scikit-learn 1.7.2, SciPy 1.14, NumPy 1.26.

Squashing a thousand fake incomes#

import numpy as np
from scipy.stats import skew
from sklearn.preprocessing import PowerTransformer

rng = np.random.default_rng(42)
# 1,000 fake monthly incomes in rupees -- most small, a few enormous
incomes = rng.lognormal(mean=10.5, sigma=0.8, size=1000)

print(f"min {incomes.min():.0f}   median {np.median(incomes):.0f}   max {incomes.max():.0f}")
print(f"skew of raw incomes: {skew(incomes):.2f}")

logged = np.log1p(incomes)                     # log(1 + x): safe even at zero
print(f"skew after log1p:    {skew(logged):.2f}")

pt = PowerTransformer()                        # Yeo-Johnson by default
squashed = pt.fit_transform(incomes.reshape(-1, 1))
print(f"skew after Yeo-Johnson: {skew(squashed.ravel()):.2f}")
print(f"fitted lambda: {pt.lambdas_[0]:.3f}")  # near 0 means 'basically a log'

min 1961   median 36495   max 461890
skew of raw incomes: 3.11
skew after log1p:    -0.04
skew after Yeo-Johnson: 0.00
fitted lambda: 0.018

The walkthrough#

The skew number is your before-and-after meter. Zero means balanced. Above about 1, the tail is long enough to matter. Raw incomes score 3.11 — the max is over 200 times the min. One log1p later, the skew is essentially gone.

log1p instead of log. np.log(0) is negative infinity, and one zero in the column poisons everything downstream. log1p computes log(1 + x), which maps 0 to 0 and behaves like plain log for large values. Make it your reflex for counts and amounts.

PowerTransformer finds the squash strength for you. It searches for lambda — the knob controlling how hard to squash — that makes the output most bell-shaped. Here it found 0.018, close to zero, which is the "use a log" setting. It also standardises its output by default, so you get shape-fixing and scaling in one step.

Yeo-Johnson vs Box-Cox. Box-Cox is the older method and demands strictly positive input. Yeo-Johnson handles zeros and negatives, which is why sklearn made it the default. If your column contains refunds or losses, Box-Cox raises an error; Yeo-Johnson does not.

Common mistakes#

Logging a column that contains zeros or negatives. log(0) is -inf, log(-5) is nan, and both flow silently into the model until an error erupts far from the cause. Use log1p for counts, Yeo-Johnson for anything that can go negative.

Forgetting to undo the transform on predictions. Train a regression on log1p(price) and the model predicts log-prices. Report np.expm1(prediction) — the exact inverse — or your "predicted price" of 11.9 will confuse everyone.

Transforming an already-balanced column. Log-transforming symmetric data creates skew in the other direction. Check skew() first; reshape only what needs it.

Fitting the transformer on all rows. PowerTransformer learns lambda from data, so it leaks like any other fitted step. Fit on training rows only, then transform the rest.

Try it yourself#

Generate rng.normal(50, 10, 1000) — already bell-shaped — and push it through the same script. Watch the skew before and after, and check what lambda the transformer picks when there is nothing to fix.

What to learn next#

• Binning and discretisation — the blunter tool: chop the column into buckets instead.
• Feature scaling — size problems, as opposed to shape problems.
• Statistics — skew, spread and distributions from the ground up.