Training error fell all the way to 60 features. Validation bottomed out at 8
You add features one at a time and record the error. Training error keeps falling. It is lowest with all sixty in. So you ship all sixty.
Validation error did something else. It bottomed out at the eighth feature and climbed from there. The sixty-feature model carries 87% more error than the eight-feature one — a fact entirely invisible from the training curve.
| Features used | Error on the training data | Error on data it has never seen |
|---|---|---|
| 8 | still has room to fall | lowest |
| 60 | lowest | 87% higher than with 8 |
The two columns point in opposite directions. Read the left one and sixty is the answer; read the right one and eight is. The right one is the column the model will actually live in.
Feature selection sounds like tidying: throw out what does not help. It is not. Every feature you add means the model spends part of your data working out what that feature does. When data is scarce, that cost comes back as worse predictions.
The traffic runs the other way too. Derived variables combine existing columns into new ones, and one well-chosen derived feature can beat twenty raw ones outright.
The key question
Does this feature improve validation performance, or only training performance?
Every feature spends some of your data
Each one adds something the model has to estimate. With unlimited data that is free; on a finite sample it means spreading the same information across more places.
Think in p over n, not p
"How many features is too many" has no fixed answer. On 40 rows the limit is around 4; on 4,000 rows it is around 14. What matters is p relative to n.
Filter
score, then cutRank each feature on its own, keep the top ones.
Wrapper
try, measure, repeatAdd or drop features and refit the model each time.
Embedded
the model decidesLasso and trees select while they fit.
| Family | Examples | Cost | Blind spot |
|---|---|---|---|
| Filter | Correlation, chi-square, mutual information, variance | Cheap — one pass | Judges each feature alone, so it misses combinations |
| Wrapper | Forward, backward, recursive elimination | Expensive — refits repeatedly | Overfits the validation set if used greedily |
| Embedded | Lasso, Elastic Net, tree importance | Free — part of fitting | Tied to that one model family |
| Domain | What the business knows is causal | Human time | Confirms what you already believed |
In practice the common combination is a filter to clear the obvious cases and an embedded method for the rest. Wrappers are expensive, and because they consult the validation set repeatedly they become a source of overfitting in their own right.
What a correlation filter cannot see
The most common selection rule is correlate each feature with the target and drop the low ones. Fast, simple, and blind to everything that is not a straight line.
| Relationship | Pearson r | A filter would | Actually |
|---|---|---|---|
| y = x, straight line | 1.00 | keep | Correct |
| y = x², U-shaped | 0.00 | drop | Perfectly predictive |
| Two clean clusters | 0.02 | drop | Separates the classes on its own |
| Predictive only above a threshold | 0.11 | drop | A tree finds it immediately |
| Useless alone, decisive with another | 0.03 | drop | An interaction term is where it lives |
| Near-duplicate of another feature | 0.94 | keep | Redundant — one of the pair is enough |
Row two is the extreme case. y = x² means x determines y exactly, and the Pearson correlation is zero. Cut on correlation and the best feature is the first one you throw away.
The last row runs the other way. It survives with r = 0.94 while carrying almost the same information as a feature you already have. Correlating with the target and adding something are different properties.
Several strongly correlated features together produce multicollinearity. Raw predictive accuracy may hold up, but the coefficients become unstable and interpretation collapses.
| Symptom | Cause | What to do |
|---|---|---|
| A coefficient with the wrong sign | Correlated features cancelling each other | Keep one of the pair |
| Coefficients swing on small data changes | The solution is barely identified | Ridge, or drop a feature |
| Significant F, no significant t | The contribution is spread across several | Diagnose with VIF |
| VIF above 10 | One feature is nearly explained by the others | Remove or combine |
| The same quantity in two units | A preprocessing slip | Delete immediately |
If one feature dominates the importance ranking, that is not a result to celebrate but a thing to check. It usually means information from after the prediction point has found its way in.
| Feature | Looks like | The problem |
|---|---|---|
| cancel_reason | A strong predictor of churn | Only filled in after the churn happens |
| total_paid including tax | Predicts order value well | It contains the target |
| account_closed_date | Useful timing information | Its presence is the label |
| A field updated nightly | Fine in the training table | Holds the future at prediction time |
| An ID that increases over time | Correlates with the outcome | It is encoding the date, not a cause |
They all share one property: each is a value that comes into existence after the outcome does. In the training table they look perfectly ordinary; at the moment you actually need a prediction, the cell is empty.
Ask the timing question
Take each feature and ask "does this value exist at the moment I have to predict?" That single question catches most leakage, and it is frequently more useful than any importance ranking.
Work in order, and select inside the fold
Before reaching for an algorithm, clear out everything you can eliminate by thinking. It is common for this pass alone to remove more than half the columns.
| Step | Remove | Criterion |
|---|---|---|
| 1 | Leaky features | The value does not exist at prediction time |
| 2 | Identifiers | IDs, names, random codes |
| 3 | Constant and near-constant | One distinct value, or 99% the same |
| 4 | Exact duplicates | The same quantity in different units |
| 5 | Review very sparse columns | Over 80% missing — keep only the indicator |
| 6 | Domain-irrelevant columns | If you cannot explain it, do not keep it |
| 7 | Run an algorithm on the rest | Lasso, tree importance, recursive elimination |
Feature selection learns from data. Select on the full dataset and then cross-validate, and each held-out fold is being scored on features chosen partly by looking at that fold.
Same data, same model
Top 20 selected on the full dataset, then cross-validated: 84.1%
Reselected within each fold: 78.6%
Actual production accuracy: 78.9%
The whole 5.5-point gap leaked in during selection. It widens as features increase and rows decrease.
A good derived feature supplies a form the model would struggle to build itself. Linear models cannot divide; trees can approximate a ratio only by spending several splits on it.
| Pattern | Example | Why it helps |
|---|---|---|
| Ratio | Spend ÷ visits = spend per visit | Removes size so behaviour shows through |
| Difference | This month − last month | Trend beats level for most decisions |
| Elapsed time | Days since last purchase | A date is not a number; an interval is |
| Aggregate | Mean, max, count over a window | One row per entity instead of many |
| Interaction | Price × season, tier × tenure | Effects that exist only in combination |
| Cyclical | sin and cos of hour or month | Keeps midnight adjacent to 23:00 |
| Flag | Was this field blank? | Missingness is often the signal |
Why ratios pull so much weight
Give a model revenue and visit count separately and it finds large customers. Give it revenue per visit and it finds high-value customers. The second is usually the question, and a linear model cannot express it from the first two.
One derived feature beating twenty raw ones
Churn prediction for a subscription product. The raw features were twenty columns pulled from usage logs — active days per month, total session time, per-feature usage counts.
| Feature set | Count | Validation AUC | Interpretability |
|---|---|---|---|
| All 20 raw | 20 | 0.734 | Low |
| Top 8 of the raw | 8 | 0.741 | Moderate |
| Last 30 days ÷ prior 30 days (usage ratio) | 1 | 0.768 | High |
| That ratio plus the top 5 | 6 | 0.803 | High |
Row three is the finding: one feature beats all twenty raw ones. What drives churn is not the level of usage but the change in it, and every raw feature captured only the level.
Could a tree have found that ratio itself? In principle. But approximating a division with splits burns depth, and depth costs data. A human writing one line is far cheaper.
The lesson here
Feature design often moves performance more than feature selection does. And design does not come from an algorithm — it comes from understanding what causes the thing you are predicting.
Importance on the same model put support_ticket_count in first place by a wide margin, and performance backed it up. Checking the timing turned up a problem.
Checking the timing
The count in training: includes the month before cancellation
The moment of prediction in production: churn status still unknown
Customers who have decided to leave file a burst of tickets on the way out. The feature was a consequence of churn, not a cause — and at prediction time it had not happened yet.
Recomputing it as "the 30 days before the prediction date" rather than "the last 30 days" dropped it to seventh in importance and took validation AUC from 0.83 to 0.80. The drop is the correct outcome. The 0.83 was performance that did not exist.
Misconception 1
❌ More features is better — the model will sort it out.
True with unlimited data. On a finite sample every feature spends part of it.Sixty features on forty rows drives training error to zero and nothing else.
Misconception 2
❌ Low correlation with the target means it can go.
Correlation sees straight lines only. U-shaped relationships, threshold effects and features that matter only in combination all come back near zero.
Misconception 3
❌ Selection is preprocessing, so do it once.
Selection learns from data. Under cross-validation it has to be redone inside every fold, or the score comes out optimistic.
Misconception 4
❌ The top-ranked feature is the most important one.
Rule out leakage first. And tree importance systematically favours high-cardinality features, so cross-check with permutation importance.
Misconception 5
❌ Generating lots of derived features costs nothing.
All pairwise ratios of twenty features is 380 columns, nearly all noise. A derived feature is still a feature and pays exactly the same price.
Add features and watch the two errors diverge
Training and validation error plotted together as features go from 1 to 60. Only five of them carry any signal; the rest is noise. Use the slider to change how much data you have.
Five features actually carry signal. Everything after that is noise the model can still memorise.
Rows
200
Best feature count
8
Cost of using all 60
+87%
With 200 rows the validation error bottoms out at 8 features and rises after that, even though training error keeps falling all the way to 60. Using every feature costs 87% more error than stopping at the bottom.
"Too many features" is not a fixed number — it is a number relative to how many rows you have.
Learning points
Training error always falls as features are added, which is why it decides nothing.
The minimum of the validation curve moves with the amount of data.
With plenty of data a useless feature costs little; with little data it is decisive.
Fewer, not more
Every feature costs data
Better, not just fewer
A ratio can beat both its parts
Select honestly
Selection is part of the model
The question is not "which features should I keep?"
It is "does this feature exist at prediction time?" The best-performing feature turns out to be a feature that does not existsurprisingly often.
Selecting and designing features moves performance more than choosing a model does. In SKARI you can check the following alongside.
Selection
Lasso, RFE, Mutual Information, Boruta
Redundancy
VIF, Correlation Heatmap, Condition Index
Importance
Permutation Importance, SHAP, Tree Importance
Dimension reduction
PCA, Factor Analysis, UMAP
Once this clicks, you can answer questions like these.