Missing Values

Three reasons, one distinction

Donald Rubin's taxonomy is still where every discussion of missing data starts. The names are forbidding; the question behind them is not — what is the blankness related to?

• •
  MCAR — missing completely at random. Blankness relates to nothing in the table. A sensor dropped one reading.
• •
  MAR — missing at random. Blankness is explained by another column you observed. One age group skipped a question.
• •
  MNAR — missing not at random. Blankness is caused by the hidden value itself. High earners will not state their income.

Why the distinction earns its keep

The final row is the fundamental difficulty of the whole topic. You cannot establish MNAR from the data. Doing so would require knowing the values that are missing — and if you knew them, they would not be missing.

So in practice you reason about it from domain knowledge. A non-response rate on the income question that is triple every other question is already grounds for suspecting MNAR.

Deleting rows is sometimes the worst option

Dropping rows with blanks is the simplest treatment and by far the most common. Under MCAR it costs sample size and introduces no bias. The trouble starts everywhere else.

That last row catches people out constantly. Five per cent per column sounds harmless, but spread independently across twenty columns only 0.95²⁰ ≈ 36% of rows are complete. Looking at per-column rates gives you no warning that this is happening.

Filling is not free either

Mean-filling keeps every row, so it looks costless. What it actually does is manufacture confidence. Every filled cell lands exactly on the mean, which makes the data look tighter than it is.

The standard error column is the one that matters. Any single-imputation method makes standard errors come out too small, because the filled-in values get treated as though they had been observed. Confidence intervals narrow, p-values shrink, and things that were not significant start looking as though they are.

Blankness is itself data

An empty "spouse's income" probably means unmarried. An empty "cancellation reason" means they have not cancelled. In cases like these the blank is not noise — it is the strongest predictor in the table.

Which is why adding an indicator column before you fill anything is nearly always worth it. One extra column buys the model access to the fact that a value was absent.

1. Draw the map before choosing anything

Look at where the blanks are before deciding what to do about them. A table of per-column rates is not enough; you need a row-by-column picture before a pattern appears.

• •
  Evenly scattered — plausibly MCAR. Delete or fill simply; either is fine.
• •
  Whole rows blank from a point onwards — abandoned responses. Treat the tail of the form together.
• •
  A vertical block — the field was not collected during that period. A date variable explains it.
• •
  Several columns blank together — one page or section was skipped. Handle them as a group.
• •
  One column far emptier than the rest — a sensitive question. Start suspecting MNAR.

2. Rough thresholds by rate

These are starting points, not rules. A column that is 60% missing can be worth keeping if it is decisive when present, and a column that is 3% missing deserves care if those 3% are MNAR.

3. Choosing a method

4. Rules that do not bend

• •
  Fit imputation parameters on the training rows only — means, medians, regression coefficients, all of them. Fitting on everything is leakage.
• •
  Never impute the target — inventing the value you are trying to predict makes that row unusable for training.
• •
  Create the indicator before filling — once filled, there is no way to recover which cells were blank.
• •
  In time series, never fill from the future — backward fill leaks tomorrow into today by construction.
• •
  Compare distributions before and after — if the shape visibly changed, the method was wrong for this column.

In practice: what mean-filling did to the p-value

A salary table for 800 employees is missing 120 salaries (15%). Fill them with the mean, then run the department comparison.

The mean held and everything else moved. Putting 120 people exactly on the mean makes the data look tighter than it is, which shrinks the p-value and narrows the interval. Confidence appeared out of nowhere.

Checking who the 120 were made it worse: mostly executives and new hires. Executive pay was withheld; new hires had not been finalised. Two groups nowhere near the mean, all assigned the mean.

In practice: when the blank was the best feature

On a loan underwriting model, previous_employment_length came in 34% missing. Before deciding whether to drop or fill it, somebody flagged the blanks and looked at them on their own.

Had the column been dropped on the grounds that 34% is too much, the model would have lost its strongest feature. Find out what the blank means before deciding what to do with it.

Common misunderstandings

Hold the rate fixed and change only the reason

A 40 × 6 map. Red cells are blank, and all three mechanisms are set to the same 12% rate. The first column is income; the cards below compare the true mean against the mean of whatever survived.

What to look for

• •
  Under MCAR, is there any structure to where the red cells land?
• •
  Why do the MAR blanks cluster in the bottom right?
• •
  MAR is clearly not random — so why is the income mean unbiased?
• •
  Under MNAR, can you see which end of the income column is empty?
• •
  Could any imputation method undo the MNAR bias?