Outlier Detection & Treatment: Z-Score vs IQR

An outlier is a value that sits far from the rest. It might be a typo, a broken sensor — or the single most important record in your dataset. The skill isn't finding them; it's deciding which is which.

Watch out

Deleting outliers by reflex is the most common mistake in data cleaning. Sometimes the outlier is the finding — fraud, a churn risk, a breakthrough result.

Real Extreme vs Error

Error: age = 999, a negative price, a duplicated row — fix or remove
Genuine extreme: a whale customer, a market crash, a rare disease — keep and study
The only way to tell them apart is context, not the number alone

How to Detect Outliers

Z-Score

Measures how many standard deviations a point is from the mean. Flag values beyond a threshold (commonly |z| > 3). Simple, but the mean and standard deviation are themselves pulled by outliers — best for roughly normal data.

IQR (Interquartile Range)

Flags anything below Q1 − 1.5×IQR or above Q3 + 1.5×IQR. Based on quartiles, so it resists skew and extreme values — a safer default for messy data. This is exactly what the box plot above shows.

MAD (Median Absolute Deviation)

The most robust option; uses the median throughout, so a few extreme points barely move it.

Method	Robust to skew?	Best for
Z-Score	No	Roughly normal data
IQR (1.5×)	Yes	Skewed / messy data — good default
MAD	Very	Heavy tails, many extremes

How to Treat Them

Remove: only for confirmed errors
Cap / winsorize: clip extreme values to a percentile — keeps the row, tames the value
Transform: a log transform pulls a long right tail back in
Keep: when the extreme is the very thing you're studying

Common Mistakes

Deleting outliers just to make a chart look nicer
Using mean-and-std detection on skewed data (the outliers hide themselves)
Removing extremes before checking whether they're the signal

Tip

When outliers are real but disruptive, prefer robust scaling or a transform over deletion — you keep the record without letting it dominate.

Outliers in the SKARI Data Editor

In SKARI's Data Editor, the outlier tool lets you pick a method — Z-Score or IQR — set the threshold, and preview the bounds before you touch a single value.

Choose Z-Score or IQR and see exactly which rows get flagged
Review the computed bounds before removing or capping anything
The action is recorded in the Pipeline history for reproducibility

Takeaway

You decide with the bounds in front of you — no blind deletion, and a fully documented step.

Frequently Asked Questions

Z-Score or IQR?

IQR by default — it doesn't rely on the mean and handles skew. Reach for Z-Score only when the data is roughly normal.

Should I always remove outliers?

No. Remove confirmed errors; for genuine extremes, cap, transform, or keep them depending on your goal.

How do outliers relate to scaling?

They distort Min-Max and Z-Score scaling. If you can't remove them, use robust scaling, which is built to withstand them.

Conclusion

Outliers deserve a decision, not a delete key. Detect them with a method that fits your distribution, judge each in context, and treat them in a way that preserves real signal.

Takeaway

Handle extremes deliberately and your models see the truth — including the rare, valuable cases.

Handling Missing Data

The other half of clean data

Data Normalization Guide

Why robust scaling exists — outliers

Advanced Clustering Techniques

Density methods that treat outliers as noise