Exploratory Data Analysis (EDA)

Two modes, and why they do not mix

Statistical work runs in two opposite modes. One generates hypotheses; the other tests them. Run them at once and you lose both.

This is why a p-value found while exploring cannot simply be reported. If you looked at a hundred relationships and picked the most striking one, that p-value has already been through a hundred rounds of selection. Exploration produces candidates; judgement has to come from somewhere else.

Four passes over the table

There is no fixed procedure, but the order matters — each pass is only worth doing once the pass before it has come back clean.

Why it leans on pictures

EDA is visual by necessity, not by taste. A summary statistic is a device for collapsing many values into one, and problems hide inside what got collapsed. Building datasets that share a mean, a standard deviation and a correlation while looking nothing alike is not even hard.

A plot, by contrast, shows you things you did not think to ask for: a second hump, values piling up on round numbers, a series that simply stops after some date. None of those can be requested from a statistic; all of them are obvious on sight.

The bill arrives later, with interest

Skip EDA and the code still runs, the model still trains, the numbers still come out. Whatever is wrong goes wrong quietly, and usually surfaces at the worst possible moment.

Assumptions can only be checked by looking

Nearly every technique comes with conditions attached, and there is no way to know whether they hold without inspecting the data.

• •
  t-tests and ANOVA — normality and equal variances. A histogram and a box plot per group settle it.
• •
  Linear regression — linearity and constant error variance. Invisible until you plot the residuals.
• •
  Correlation — the relationship has to be a straight line. A clean curve reports r near zero.
• •
  k-means — roughly spherical clusters of similar size. A scatter plot tells you in seconds.
• •
  Time-series models — stationarity. One plot against time exposes trend and seasonality.

You learn what the model is about to learn

EDA is not purely defensive. Which features will matter, what needs transforming, which interactions to include — most of that gets decided here. Expecting the model to work it out on its own is usually optimistic.

1. The first half hour

Run these in this order on any table you have not seen before. Most disasters get caught somewhere in these eight lines.

2. Start with the min and the max

For a numeric column these two numbers earn their keep faster than anything else. A mean can hide almost any problem; the extremes cannot.

3. Count the distinct values in text columns

Categorical problems are almost always the same value spelled several ways, and a distinct count exposes them instantly. If a column that should hold fifty states holds a hundred and twelve, you already have your answer.

4. Missingness is a pattern, not a percentage

"8% missing" is far less informative than where those 8% sit. Scattered evenly, most imputation methods are harmless. Concentrated in one segment, imputing manufactures facts that were never collected.

5. Keep exploring and testing apart

The better your EDA, the less entitled you are to test on the same rows. There is exactly one way out of that bind: split off a portion of the data before you start looking, and touch it only at the end.

• •
  Hold out a test set before the first plot
• •
  Write down each hypothesis as it occurs to you
• •
  Test only the written-down ones, only on the held-out data
• •
  Never report a p-value you found while browsing
• •
  Log which transformations you applied and why

In practice: day one on a churn model

You have been handed churn prediction for a subscription product. The data team sends a CSV: fifty thousand rows, twenty-eight columns. Working through the checklist turns up the following.

Row six is the day's real find. cancel_reason only gets populated after somebody churns, so including it pushes validation accuracy past 99% — and makes the model worthless in production, where that field is still empty at the moment you need a prediction.

In practice: one plot, opposite conclusion

The same dataset showed that higher monthly fees came with lower churn. A proposal to raise prices was nearly written. Splitting the plot by tier told a different story.

Read the correlation on its own and you land on a price increase — precisely the wrong move. Skipping the fourth pass, where you split by a third variable and look again, is how that happens.

Common misunderstandings

The same 200 rows, seen four ways

One customer table. The three numbers at the top — row count, mean visits, mean revenue — never change as you step through. Each pass, meanwhile, turns up something the previous one had no way of showing you.

What to look for

• •
  In pass 1, why does it matter that signup arrived as text?
• •
  In pass 2, how far does the mean age fall once the 999 bar is removed?
• •
  In pass 3, what would make the points fall into two separate clouds?
• •
  In pass 4, what does an r near zero for free users actually tell you?
• •
  If you had only reported the pooled r = 0.84, what would you have concluded?