Handling Missing Data

Almost every real dataset has holes in it — a blank survey answer, a sensor that dropped out, a field no one filled in. How you handle those gaps quietly shapes every result that follows.

Why Missing Data Matters

• Bias: if the missing rows differ from the rest, dropping them skews your sample
• Lost power: fewer rows means weaker, less reliable estimates
• Broken models: many algorithms simply refuse to run with gaps

Three Types of Missingness

MCAR — Missing Completely At Random

The gap has nothing to do with any value. A lab sample is dropped by accident. Safe to delete or impute.

MAR — Missing At Random

Missingness depends on other observed columns — e.g. older respondents skip an income question. Imputation using the related columns works well.

MNAR — Missing Not At Random

The gap depends on the missing value itself — high earners hide their income. The hardest case; deletion and naive imputation both bias the result.

How to Handle It

Deletion

• Listwise (drop the row): fine when gaps are few and MCAR
• Drop the column: when a field is mostly empty and low-value

Imputation

• Mean / median: quick; median resists skew and outliers
• Mode: for categorical fields
• Interpolation, forward/back fill: for ordered or time-series data

Common Mistakes

• Mean-imputing a skewed column pulls it toward outliers — use the median
• Filling MNAR gaps with a mean hides the very signal you care about
• Silently dropping rows without checking whether they differ from the rest

Missing Data in the SKARI Data Editor

SKARI's Data Editor makes this visual and reproducible. Open the Missing Heatmap to see exactly where and how gaps cluster, then apply Fill Missing with the method that fits.

• Missing Heatmap: spot whether gaps are scattered (MCAR) or patterned (MAR/MNAR)
• Fill Missing: mean, median, mode, and interpolation in a click
• Every step is logged in the Pipeline history, so the cleaning is reproducible

Frequently Asked Questions

Mean or median imputation?

Median by default — it isn't dragged around by outliers or skew. Use the mean only for roughly symmetric data.

Is dropping rows ever fine?

Yes, when gaps are few and completely random (MCAR). Check first that the dropped rows don't differ systematically from the rest.

Can tree models handle missing values?

Some (like XGBoost) have built-in handling, but explicit, documented imputation is usually clearer and safer.

Conclusion

Missing data isn't a nuisance to delete on autopilot — it's a signal. Diagnose why it's missing, pick deletion or imputation to match, and keep the test set untouched until the end.