Concept·

Categorical Encoding

A 50% churn rate from 12 customers went into the model as a fact

To get a city column into a churn model, you use target encoding: replace each city with that city's churn rate. Validation accuracy jumps.

But the data holds only 12 customers from Jeju, six of whom churned. The encoded value is 0.500. That number is not a fact about Jeju — it is a transcript of which of those twelve churned, and the model is using it to predict them.

There are several ways to turn a city name into a number, and each of them makes Jeju a different number — each telling the model a different story.

How the city column becomes numbers	What Jeju becomes	What that number tells the model
Number the cities (Seoul 1, Busan 2, Jeju 3)	3	Jeju is larger than Busan and three times Seoul
One 0/1 column per city	Jeju column = 1, rest = 0	the cities have no order and no size
Replace it with that city's churn rate	0.500	six of Jeju's twelve churned — the answers, written down

Categorical encoding gets treated as a mechanical step for turning text into numbers. What it really does is write a claim about how the categories relate into the data — and if the claim is wrong, the model takes it as true.

The key question

What is this encoding asserting about the categories?

Concept

Assigning a number makes a claim

An encoding is an assertion

Assign Seoul 0, Busan 1 and Jeju 5 and the column is now numeric — and numbers bring order and spacing with them. Linear regression reads "Jeju is five more of something than Seoul"; kNN reads "Jeju is nearer Daejeon than Seoul".

Label

invents an order

Fine for ordinal data, wrong for nominal.

One-hot

one column each

No fake order, but the width grows with categories.

Target

one column, high risk

Strong signal, and the fastest route to leakage.

Frequency

one column, no leak

Encodes how common a category is, nothing else.

So the first question never changes: do these categories actually have an order?If they do, preserve it. If they do not, do not invent one.

Column	Order	Suitable encoding
Blood type (A/B/O/AB)	None	One-hot
Tier (basic/silver/gold)	Yes	Order-preserving integer
Day of week	Cyclical	Sine and cosine, or one-hot
Postal code	None	Target or frequency (high cardinality)
Satisfaction (1–5)	Yes	Already an integer — leave it
Product SKU	None	Embedding or hashing

The main encodings

Encoding	Columns	Use for	Danger
One-hot	k	Nominal, few categories, linear models	Explodes with high cardinality
Dummy (k−1)	k − 1	Regression, where you need an intercept	The dropped level becomes the baseline
Ordinal / label	1	Genuinely ordered categories	Invents an order on nominal data
Target / mean	1	High cardinality, tree models	Leaks the outcome without care
Frequency / count	1	When rarity itself is informative	Two categories can share a count
Hashing	fixed	Streaming, unseen categories	Collisions merge unrelated categories
Embedding	d	Very high cardinality, neural nets	Needs a lot of data to learn

One-hot versus dummy

One-hot gives k columns for k categories; dummy coding gives k − 1. In regression you need the dummy version: the k columns always sum to 1, which duplicates the intercept exactly. This is the dummy variable trap.

\text{Seoul} + \text{Busan} + \cdots + \text{Jeju} = 1 = \text{intercept}

•
Linear and logistic regression — k − 1, with one level held out as the baseline.
•
Ridge and Lasso — k is fine; the penalty absorbs the collinearity.
•
Tree models — k. There is no intercept, so there is no trap.
•
Neural networks — k. The bias term handles it.

Why It Matters

Different models want different encodings

The right encoding depends on the model

Model	One-hot	Label	Target	Note
Linear / logistic regression	✓	✗	△	Label encoding forces a straight-line effect
Decision tree, random forest	△	△	✓	Wide one-hot weakens each split
Gradient boosting	△	△	✓	Many libraries handle categories natively
kNN, k-means, SVM	✓	✗	△	Label encoding distorts every distance
Neural networks	△	✗	△	Embeddings beat both above a few hundred levels

Row two gets misread often. Trees can tolerate label encoding, because enough successive splits can carve out any grouping of categories. But it burns depth doing so, which is why target encoding is far more efficient once there are many categories.

High cardinality breaks one-hot

One-hot is safe and adds a column per category. With postal codes or product codes running into the thousands, most of those columns are almost always zero.

Categories	One-hot columns	Problem
5	5	None
50	50	Starting to strain a linear model
500	500	Tree splits scatter across individual levels
5,000	5,000	Columns can outnumber training rows
50,000	50,000	Impractical on both memory and time

Which is why high cardinality moves you to target encoding, embeddings or hashing— three ways of compressing the same information into one column or a handful.

Exactly why target encoding is dangerous

Target encoding replaces each category with the mean outcome for that category. Powerful, and by construction a feature built by looking at the answer.

\text{enc}(c) = \frac{n_c \bar{y}_c + m \bar{y}}{n_c + m}

Without smoothing ( $m = 0$ ) each category simply becomes its own group mean. The fewer rows in the group, the more that mean is those rows' own answers. An encoded value of 0.500 for a twelve-row category is essentially an answer key for those twelve rows.

Jeju's encoded value (overall churn 0.039)

m = 0 → 0.500 (6 of 12 churned)

m = 20 → 0.212

m = 100 → 0.089

Under the same settings Seoul, with 4,200 rows, goes 0.031 → 0.031 and barely moves. Smoothing pulls only the categories that lack the data to speak for themselves.

What leakage looks like from the outside

If adding target encoding makes the validation score jump noticeably, the likeliest explanation is not a better feature but a leak. Redo it with out-of-fold encoding and the gain usually evaporates.

How It Works

Order, cardinality, model — in that order

1. Three questions

•
Is there an order? If so, an order-preserving integer. If not, next question.
•
How many levels? Up to roughly fifteen, one-hot. More than that, next question.
•
Which model? Trees take target encoding, neural nets take embeddings, streaming takes hashing.

Asked in that order, most columns resolve to a single answer. When it is genuinely unclear, start with one-hot — it may be slow, but it will not be wrong.

2. Guards for target encoding

▼ Table 1 Five ways to keep target encoding from leaking

Guard	What it does	When to use it
Fit on training rows only	The category means never see the test set	Always — this one is not optional
Smoothing	Pulls small groups toward the overall rate	Whenever some categories are rare
Out-of-fold encoding	Each row is encoded from the other folds	Any cross-validated pipeline
Adding noise	Blurs the encoded value slightly	When overfitting persists after smoothing
Group rare levels	Everything below a threshold becomes "other"	Long tails of one-off categories

The first and third rows are not optional. Category means come from the training rows only, and inside cross-validation each row must be encoded from the folds it is not in.

3. Categories you have never seen

This one arrives in production without fail. A city that was not in training shows up and the encoder has no value for it. Decide in advance or it throws at request time.

Encoding	Handling an unseen level	Note
One-hot	A row of all zeros	Safer still to train an explicit "other" column
Ordinal	Most frequent level, or missing	A middle rank has nothing to justify it
Target	The overall mean	Falls out of the smoothing formula naturally
Frequency	Zero, or the minimum count	Never seen is literally a count of zero
Hashing	Handled automatically	The main reason to reach for hashing

4. Grouping rare levels

Hundreds of categories that appear once each are noise given its own columns. Set a threshold and collapse them into "other".

•
Below 1% of rows is the usual cut
•
An absolute count (say, under 30) is clearer in sample-size terms
•
Set the threshold on training rows only — reading test frequencies leaks
•
If "other" ends up huge, lower the threshold
•
Keep a small category that matters to the domain

Cyclical categories need their own treatment

Days, months and hours wrap around. Sunday (6) and Monday (0) are the two furthest values under label encoding and adjacent in reality. Encoding them as a sine and a cosine pair preserves that adjacency.

Example

How target encoding inflated a validation score

In practice: a validation score that went up and came back

A record of putting one city column (87 levels) into a churn model several ways. Same model, same data, encoding only.

Encoding	Columns	Validation AUC	Production AUC	Read
Dropped	0	0.742	0.739	Baseline
One-hot	87	0.761	0.758	A real gain, but wide
Label	1	0.744	0.741	Effectively nothing
Target (unsmoothed)	1	0.883	0.752	Leakage
Target (m=50, out-of-fold)	1	0.769	0.766	The best option
Frequency	1	0.751	0.749	A small gain

Row four is the trap. A validation AUC of 0.883 is far above anything else on the list — and production comes in at 0.752. That 0.13 gap is entirely the price of letting the model see the answers.

Row five is the one to ship. Its validation score is much lower than row four's, yet production is higher — and, more importantly, validation predicts production almost exactly.

The lesson here

Never compare encodings on the validation score alone. The gap between validation and production is the more informative number, and a large gap means the encoding has been looking at the target.

In practice: throwing away an order

A customer tier (basic, silver, gold, platinum) was one-hot encoded. It looked like the safe choice and performance went down.

Same variable, two encodings

One-hot, 4 columns: validation AUC 0.771

Order-preserving integer, 1 column: validation AUC 0.784

The tiers genuinely are ordered. One-hot deletes the fact that gold outranks silver, so the model has to relearn that relationship from the data — and with limited data it never quite does.

One-hot is not a universally safe default. On ordered categories it is the option that discards information.

Common misunderstandings

Misconception 1

❌ Numbering categories is just a format change.

Numbers carry order and spacing. Seoul as 0 and Jeju as 5 tells the model there are five units of something between them. That is not a change of format but a change of content.

Misconception 2

❌ One-hot is always the safe choice.

On ordered categories it throws the order away. And at a few thousand levels the columns outnumber the rows, which makes training impossible rather than merely slow.

Misconception 3

❌ Target encoding raised the score, so it is a good feature.

Without smoothing and out-of-fold construction, the model has seen the answers. Up in validation and down in production is the signature of exactly that.

Misconception 4

❌ Encoding before the split saves time.

One-hot and frequency encoding are broadly fine. Target encoding never is — the category means would contain the test set's own labels.

Misconception 5

❌ Day of week can go in as 0–6.

That makes Sunday and Monday the two most distant values when they are adjacent days. Cyclical categories need a sine/cosine pair, or one-hot.

Interactive

One column, four encodings

A city column with six levels. Jeju has just 12 rows and a churn rate recorded as 50%. Watch how each encoding treats that row.

One city column with six categories, encoded four ways. Jeju has only 12 rows.

Label — one integer per category

Category	n	Churn	Encoded
Seoul	4,200	3.1%	0
Busan	1,100	4.8%	1
Incheon	900	4.2%	2
Daegu	620	5.5%	3
Daejeon	380	6.1%	4
Jeju	12	50.0%	5

Columns added

Jeju encoded value

Seoul becomes 0 and Jeju becomes 5, so any model that treats the column as a number now believes Jeju is five units more of something than Seoul. Nothing in the data says that.

An encoding is a claim about the categories. Make sure it is a claim you meant.

What to look for

•
Under label encoding, what does the gap between Seoul and Jeju come to mean?
•
How many columns does one-hot produce? What if there were 500 levels?
•
Under target encoding, what is Jeju's 0.500 a transcript of?
•
Raise the smoothing weight — does Jeju or Seoul move more?
•
Why is frequency encoding incapable of leaking?

Learning points

Label encoding manufactures an order that was not there.

One-hot is safe and costs a column per category.

Target encoding is strong because it looked at the target — and risky for the same reason.

Smoothing pulls only the categories that lack the data to stand on their own.

Key takeaways

Order or not

The first question, before anything else

• Nominal categories must not be numbered
• Ordinal categories should keep their order
• One-hot on ordinal throws the order away

Width or leakage

One column per category, or one clever column

• One-hot is safe and wide
• Target encoding is narrow and risky
• Frequency encoding is narrow and safe

Rare levels

Small groups produce confident nonsense

• 12 rows can give a rate of 0.500
• Smooth toward the overall rate
• Or group them into "other"

The question is not "which encoding scores best?"

It is "what is this encoding asserting about the categories?" An invented order, or a peek at the target, is an assertion the model will accept as true.

How does this apply to real data?

Encoding comes after the train/test split, and putting it in a pipeline means unseen categories get handled the same way every time. In SKARI you can check the following alongside.

Encoders

One-Hot, Ordinal, Target, Frequency, Hashing

Category diagnostics

Cardinality, Rare Level Scan, Crosstab

Categorical tests

Chi-square, Cramér's V, Fisher Exact

Leakage checks

Out-of-Fold Encoding, Pipeline, Leakage Check

Once this clicks, you can answer questions like these.

•
Do these categories genuinely have an order?
•
How many levels are there, and can one-hot carry that?
•
Is this encoding looking at the target?
•
Where should the cut for rare levels sit?
•
What value does an unseen category get in production?

Now try it on real dataOpen in Lab

Go Deeper

Is this column nominal or ordinal?

Measurement levels

What if one category is spelled several ways?

Data cleaning

Do encoded columns need scaling too?

Scaling

Feature Selection & Derived Variables