WHAT IT IS
Set the outcome to predict, then
compare how several models perform.
Name what you want to predict and several models are trained and compared. It is for outcomes that have not happened yet — customer churn, revenue, and the like.
Choose what to predict
Pick one column from your data. Whether it is numeric or categorical sets the type of analysis, and the remaining columns are used for the prediction.
Set how far the search goes
Quick, Standard or Thorough. A longer search looks at more models and combinations, though the score does not necessarily improve with it.
Compare the models’ results
Every trained model can be compared on the leaderboard. Read the score together with how it was measured, and pick the model that suits your data and purpose.
THE PROBLEM
A score can change depending on
which data it was measured on.
Measuring performance on the same data used for training can give a score higher than the real one. To see how well it predicts on new data, evaluation rows have to be set aside.
Measuring on the training data alone
Performance measured on the rows used to search can look better than it is. It has to be checked on data that was not used for the search.
One person’s records split across both
When one person’s records land in both the training and the evaluation rows, the score can be inflated. Splitting by group keeps each person on one side.
A score on its own is not enough
A metric like AUC does not tell you which cases to look at first in practice. Read the metric together with the predictions and what you mean to use them for.
HOW IT IS MEASURED
Performance is checked on data the
model search never used.
Part of the data is kept aside for evaluation. It is not used for the model search or for training; once the search is over, it is used to check performance. The result is a reference for how the model may do on new data.
Leaderboard · the search score and the evaluation score
| Model | Search score | Final score | vs 1st |
|---|---|---|---|
| Weighted ensemblecombined | 0.571 | 0.603 | — |
| LightGBMsingle | 0.538 | 0.587 | −.033 |
| CatBoostsingle | 0.524 | 0.571 | −.047 |
| Random forestsingle | 0.461 | 0.498 | −.110 |
Scored separately on the evaluation rowsYes · 296 rows
Gap between the search and evaluation scoresNo · 0.032
Reading the two scores together shows how far the search result and the evaluation result differ. A large gap is worth looking into — overfitting, or how the data was split.
SPLITTING THE EVALUATION ROWS
Split the evaluation rows to match the
shape of your data and your situation.
When the outcome follows a date, or one person has several records, splitting by time or by group tends to suit better.
Setup · how the evaluation rows are split
Randomdefault
Splits the rows at random. For classification it stratifies, so each class keeps a similar share.
By time
Sorts by date and keeps the most recent rows for evaluation. Right when past data is used to predict what comes after.
By group
Splits by group so one person’s or one institution’s records do not land on both sides.
When the class balance shifts after the split, the share in the whole dataset and in each part is shown together. Read the evaluation result alongside the split method and the class balance.
USING THE RESULT
See how much of the real outcome a
top slice actually holds.
Sort the cases by predicted score and you can see how much of the real outcome the top slice holds. Lift and gain compare that against picking at random.
Leaderboard · lift and gain on the evaluation rows
Had they been picked at random, each slice would hold the same share.
EXPLAINING THE MODEL
Look at which variables influenced
the prediction.
You can see the variables that weighed most across all predictions, and the reasons behind an individual case. The explanation is a reference for how the model decided, not a causal claim.
Explain · the variables used most in the prediction
When two variables are close to each other, the importance can look split between them. Check how the variables relate as well.
Explain · how the prediction changes with a variable’s value
With the other conditions held fixed, this shows how the average prediction changes as the variable moves.
PREDICTING NEW DATA
Load new data into the chosen model,
read the predictions and download them.
Put new data through the model you chose and read the predictions. File prediction gives the data back with the results added, sorted by predicted score if you want.
Predict · choose how to use it
One case at a time
Support
A whole file at once
Back office
A shared link
Colleagues
API
Developers
Highest predicted scores first
FURTHER ANALYSIS
Once a model is chosen, the result can
be read from several angles.
Beyond performance, you can work out the cost, compare performance by group, line predictions up against real outcomes, and watch performance and data change over time.
Expected gain and loss, with cost
Enter the value of a correct prediction and the cost of a wrong one to work out the expected gain or loss. The result follows the assumptions you enter.
Comparing performance by group
Beyond the overall score, performance can be compared across the groups of a column you choose. Look for a group where it does worse.
Predictions against real outcomes
Once the real outcome is known, it can be compared with the prediction and performance recalculated. Results on the evaluation rows and in live use can differ.
Monitoring
Over time the input data or the predictions can change. Performance, data drift and the prediction mix are tracked and compared against a baseline.
THE ORDER OF WORK
Five steps, start to finish.
From preparing the data to predicting new data, the work runs in five steps. Each step shows the settings it needs and the results it produced.
01
Data
Load the data and choose the column to predict. The state of the data — missing values and the like — is shown too.
02
Setup
Set the search time and how the evaluation rows are split. Costs can be entered here if you need them.
03
Leaderboard
Compare the models the search produced, and how they do on the evaluation rows.
04
Explain
Look at variable importance and the reasons behind an individual prediction.
05
Deploy
Read the predictions for new data and download them as a file.
Shows the step you are on
The step bar shows where you are and what comes next.