A $2,000 pay gap outweighed eight years of service 250 to 1
You run kNN over an employee table to find whoever most resembles a given person. Two candidates come up. A differs only in salary, by $2,000, with identical service. B differs only in service, by 8 years, on identical pay.
Obviously A is the closer match. But compute a Euclidean distance on the raw numbers and this is what comes back.
| Candidate | Salary gap | Service gap | Distance on the raw numbers |
|---|---|---|---|
| A — the one a person would pick | $2,000 | none | 2,000 |
| B | none | 8 years | 8 |
A is scored 250 times further away. The algorithm returns whichever is closer, so it returns B. Salary is only bigger because it is counted in dollars — and on that basis alone, years of service stopped counting at all.
Scaling exists to deal with this. But describing it as "preprocessing that makes the units match" only gets you halfway. What it actually does is decide how much each variable counts — and different scalers decide differently.
The key question
Does this model compute distances? If so, which variable is currently dominating them?
Removing the units exposes the weights
All of them have the same shape: subtract something, divide by something. Only the two somethings differ.
Standardisation
(x − μ) / σCentre 0, SD 1. Unbounded, keeps the shape.
Min-Max
(x − min) / (max − min)Exactly 0 to 1. One outlier sets the whole range.
Robust scaling
(x − median) / IQRUses quartiles, so outliers do not set the scale.
Unit norm
x / ‖x‖Scales each row, not each column. Direction only.
| Scaler | Output range | Good for | Weak against |
|---|---|---|---|
| Standardisation | Unbounded, centred on 0 | Most models; the default choice | Outliers inflate σ |
| Min-Max | Exactly 0 to 1 | Neural nets, images, bounded inputs | One outlier compresses everything else |
| Robust scaling | Unbounded, centred on the median | Data with known outliers | Needs a meaningful IQR |
| MaxAbs | −1 to 1, zero stays zero | Sparse matrices | Still driven by the largest value |
| Unit norm | Each row has length 1 | Text vectors, cosine similarity | Discards magnitude entirely |
| Quantile / rank | Uniform or normal by construction | Badly behaved distributions | Distorts distances between values |
The most persistent confusion in the area. Standardisation shifts and stretchesa distribution and leaves its shape untouched. Standardise data with a skewness of 3 and the skewness is still 3.
| What you want | What to use | What changes |
|---|---|---|
| Put variables on comparable footing | Scaling | Centre and width |
| Straighten a skewed distribution | Log or square-root transform | The shape |
| Force everything into 0–1 | Min-max | The range (shape unchanged) |
| Make it closer to normal | Box–Cox or quantile transform | The whole distribution |
| Reduce the pull of extremes | Winsorise or robust scaling | The weight given to outliers |
When you need both
Feeding skewed data into a distance-based model calls for the transform first and the scaler second. Reverse them and the transform is applied to values that have already been shifted, which gives a different result.
Anything that measures distance depends on this
Euclidean distance squares each axis's difference and adds them up. When the axes carry different units, that sum is essentially the largest-unit axis and nothing else. The other axes may as well not exist.
| Method | Scaling | Why |
|---|---|---|
| k-means, kNN, SVM, DBSCAN | Required | Distance is computed in raw units |
| PCA | Required | Components chase whichever variable has the largest variance |
| Ridge, Lasso, Elastic Net | Required | The penalty applies to coefficient size |
| Neural networks | Required | Gradients behave badly on mismatched scales |
| Linear regression, unpenalised | Not needed | Coefficients absorb the units |
| Decision trees, random forest, boosting | Not needed | Splits depend only on order |
| Naive Bayes, chi-square | Not needed | No distance is involved |
The bottom three rows matter as much as the top four. Trees do not need scaling, because a split of the form "salary > 50,000" uses order only — apply any monotonic transform and you get exactly the same split.
Ridge and Lasso penalise the size of the coefficients — and coefficient size depends on the variable's units. Measure salary in dollars rather than thousands and the coefficient becomes tiny, so the penalty barely touches it.
Same variable, different units
Salary in dollars: coefficient 0.0031 → effectively unpenalised
Salary in thousands: coefficient 3.1 → properly penalised
The model has not changed, yet units decide which variables survive. An unscaled Lasso drops large-unit variables first.
Principal components find the direction of maximum variance. If salary has a variance a million times larger than years of service, the first component is the salary axis, and nothing else contributes.
This is why PCA distinguishes between running on the correlation matrix and on the covariance matrix. Correlation-based PCA is precisely PCA after standardising.
There is a PCA that should not be scaled
When every variable shares a unit and the differences in variance are themselves the signal, leave them alone. Thirty-two channels from the same sensor array are the classic case: which channel moves most is the information.
Pick one of four, and fit it on training rows only
| Situation | Use | Why |
|---|---|---|
| No particular reason to do otherwise | Standardisation | The most reliable default |
| Neural network inputs | Min-max or standardisation | Matches the working range of activations |
| Known outliers in the data | Robust scaling | Keeps outliers from setting the scale |
| Sparse matrices (mostly zeros) | MaxAbs | Zero stays zero, so sparsity survives |
| Text embeddings, cosine similarity | Unit norm | Only the direction is being compared |
| A badly misbehaved distribution | Quantile transform | Keeps the ranks, forces the shape |
A scaler learns parameters. The mean, the SD, the minimum, the maximum, the median — every one of them is a number extracted from data. The moment those numbers have seen the test set, you have leakage.
| Step | Correct | Leaky |
|---|---|---|
| Compute μ and σ | From the training rows only | From the whole dataset |
| Transform the training set | fit_transform | fit_transform |
| Transform the test set | transform | fit_transform |
| Inside cross-validation | Refit on each fold | Fit once, outside the loop |
| At serving time | Reuse the stored training μ and σ | Recompute on the incoming batch |
Row three is the one that goes wrong most often. The test set should get transform alone; call fit_transform and it is standardised against its own mean, which flatters the score.
| Column type | Treatment | Note |
|---|---|---|
| One-hot dummies | Usually leave alone | Already 0/1, so no scale problem |
| Ordinal integers (1, 2, 3) | Standardising is defensible | Only if equal spacing is acceptable |
| Binary 0/1 | Leave alone | Scaling only makes it harder to read |
| Counts | Transform, then scale | Log or square root comes first |
| Already a proportion (0–1) | Usually leave alone | Unnecessary if the other variables are comparable |
One scaler, completely different clusters
Customers clustered with k-means on age, annual spend and purchase frequency. Changing nothing but the scaler produced entirely different segments.
| Scaler | What separates the clusters | Largest cluster | Read |
|---|---|---|---|
| None | Spend, and nothing else | 78% | Effectively a three-way split on spend |
| Min-max | Mostly spend | 61% | One top spender sets the scale |
| Standardisation | All three fairly evenly | 41% | Segments you can actually describe |
| Robust scaling | All three fairly evenly | 38% | Similar to standardisation, better balanced |
Same algorithm, same data, same k. One line of preprocessing changed — and the answer to "how do our customers divide up" changed with it.
The lesson here
Any report of a clustering result has to state which scaler produced it. A scaler is not invisible plumbing; it is the modelling decision that defines what counts as similar.
A classifier cross-validated at 91.2% accuracy and ran at roughly 88% once deployed. The code turned out to be scaling outside the cross-validation loop.
Same data, same model
Scaler fitted outside the loop: CV accuracy 91.2%
Refitted on every fold: CV accuracy 88.4%
Actual production accuracy: 88.1%
The second figure predicted production almost exactly. The first was a three-point illusion created by letting each held-out fold contribute to the mean it was standardised against.
Even something as innocuous as standardisation leaks. The effect is small, but the direction is always optimistic, so it always shows up as a drop after deployment.
Misconception 1
❌ Scaling makes the data normal.
It changes the centre and the width and leaves the shape alone. Skewness and kurtosis come through untouched. Changing the shape requires a transform — a log, a Box–Cox.
Misconception 2
❌ Min-max is safest because it bounds everything to 0–1.
A single outlier sets that range. One person on $150,000 compresses everybody else below 0.2. And in production, values outside the training range break the 0–1 guarantee anyway.
Misconception 3
❌ It is only preprocessing, so the full dataset is fine.
A scaler learns numbers from data. Fit it on everything and the test set's mean and variance flow into training, which makes the score optimistic.
Misconception 4
❌ Scaling everything costs nothing.
On tree models it changes no result and destroys interpretability. A rule reading "salary > 50,000" becomes "salary > 0.37", which nobody can act on.
Misconception 5
❌ Which scaler you use is a matter of taste.
The scaler sets the weight of every variable. Your clusters, your kNN neighbours and the features Lasso discards all depend on it. It belongs in the writeup next to the result.
Change the scaler and watch the neighbour change
One query employee and two candidates. A differs by $2,000 in salary alone; B by 8 years of service alone. Which one is nearer is decided by the scaler.
One query employee and two candidate neighbours. Which one is "closer" is decided by the scaler, not the data.
No scaling — raw units
| Candidate | Salary | Years | Distance from Q |
|---|---|---|---|
| Q | $52,000 | 6 | — |
| A$2,000 more in salary, same years | $54,000 | 6 | 2.00e+3 |
| B8 more years, same salary | $52,000 | 14 | 8.00e+0 |
Nearest neighbour
B
How much further B is
4.0e-3×
In raw units a salary gap of $2,000 and a service gap of 8 years get added together as if they were the same kind of number. A — who differs from Q by one modest raise and nothing else — scores as 250 times more distant than B, and kNN picks B.
Choosing a scaler is choosing how much each variable counts.
Learning points
Without scaling, the largest-unit variable becomes the entire distance.
Different scalers can agree on the ranking and still assign very different weights.
With an outlier present, min-max under-weights, standardisation lands between, and robust scaling is unaffected.
It sets the weights
Scaling decides what counts as far
Scaling is not normalising
The distribution keeps its shape
Fit on training only
A scaler learns parameters
The question is not "should I scale?"
It is "how much weight does this scaler give each variable?" A scaler looks like invisible preprocessing and is really the step that defines what counts as similar.
Scaling is the first step after the train/test split, and bundling it into a pipeline makes the ordering impossible to get wrong. In SKARI you can check the following alongside.
Scalers
Standard, Min-Max, Robust, MaxAbs, Quantile
Distance-based analysis
k-means, kNN, DBSCAN, Hierarchical
Dimension reduction
PCA, t-SNE, UMAP, Factor Analysis
Leakage checks
Pipeline, CV Fold Isolation, Leakage Check
Once this clicks, you can answer questions like these.