Artificial Neural Networks (MLP)

Are customers who have used a service for only a short time more likely to cancel? Do customers with a high monthly fee cancel more often? Then how should we predict a customer whose tenure is short but who gets large discounts and uses the service a lot?

In real data, a single variable often can’t explain the outcome on its own. Several variables, such as tenure, monthly fee, contract type and usage count, can influence one another and form complex patterns.

A popular method for learning these relationships is the multilayer perceptron (MLP).

An MLP is a type of artificial neural network made up of an input layer, one or more hidden layers and an output layer. Each layer computes and transforms the information passed from the previous layer, and during training the weights are adjusted over and over so the prediction error gets smaller.

An MLP can be used not only for classification problems such as whether a customer churns, but also for regression problems that predict continuous values such as purchase amount or sales.

In this lesson we’ll go from the basic structure of an MLP through neurons and weights, activation functions, forward and backpropagation, key hyperparameters, a worked example, and evaluating predictive performance and the cautions in interpreting it.

Ordinary linear regression expresses the relationship between the input variables and the outcome as a linear equation. An MLP, in contrast, can learn more complex relationships by using hidden layers and nonlinear activation functions.

For example, suppose we want to predict whether a customer will cancel a service. If we feed in tenure, monthly fee, contract type, number of customer-service inquiries and recent usage count, the MLP can go through several layers of computation and predict that customer’s probability of cancelling.

During training, the model compares its predictions with whether customers actually cancelled to compute the error, and adjusts the weight on each connection to reduce that error.

That said, the fact that an MLP can learn complex relationships doesn’t mean it guarantees higher accuracy on every dataset. If there is little data or the relationships between variables are fairly simple, logistic regression or a tree-based model may be a better fit.

The input layer receives the variables used in the analysis, the hidden layers transform that input over several stages, and the output layer computes the final prediction.

For example, with 5 input variables we might put 16 neurons in the first hidden layer and 8 neurons in the second. In that case, each hidden-layer neuron computes a weighted sum of the values passed from the previous layer and then applies an activation function.

The structure of the output layer depends on the outcome being predicted. In binary classification, a single output neuron can estimate the probability of the event; in a regression problem, it can output a continuous predicted value. When designing an MLP, not only the number of hidden layers but also the number of neurons in each layer and the activation function are important settings.

What are neurons and weights?

In an MLP, a neuron is the basic computing unit: it takes in values, computes with them and passes the result to the next layer.

A neuron multiplies each of its inputs by a weight and adds them up. It then adds a bias and applies an activation function to compute its output. As a simple formula:

For example, suppose a neuron takes tenure, monthly fee and number of customer-service inquiries as inputs.

Assuming a bias of 0.1, the weighted sum is (-0.8 × -0.5) + (1.2 × 0.4) + (1.5 × 0.7) + 0.1 = 2.03

The neuron then applies its activation function to compute the value it passes to the next layer. The important point is that the analyst doesn’t set these weights by hand; they are estimated from the data during training.

Also, the weights in this example are only the values attached to one neuron. You shouldn’t read them like ordinary regression coefficients and use them to describe the independent effect of tenure directly.

An important reason MLPs use nonlinear activation functions is to learn complex nonlinear relationships. Even with several hidden layers, if each layer performs only linear computations, the whole computation can collapse into a single linear transformation. So simply stacking layers isn’t enough to represent complex nonlinear patterns.

Common activation functions are:

For example, the ReLU function outputs 0 when the input is -2 and 3 when the input is 3. By repeating such nonlinear transformations, an MLP can represent complex relationships and interactions among input variables. Still, the choice of activation function can affect training speed and predictive performance, so it should be made to suit the data and the model structure.

For example, suppose a customer has a tenure of 8 months, pays a monthly fee of $140 and used the service 3 times in the past month. The MLP first takes in these values. The first hidden layer then performs a number of weighted sums and activation-function computations and passes the results on to the next hidden layer.

Finally, the output layer computes the customer’s probability of cancelling the service. For example, if the output is 0.72, a model built to be read as a probability can express this customer’s predicted churn probability as 72%. But to know whether the prediction at this stage is accurate, we have to compare it with whether the customer actually cancelled.

During training, forward propagation computes the predictions, and a loss function then measures the prediction error. So forward propagation is the process of making predictions with the model’s current weights, while training also includes adjusting the weights using the difference between those predictions and the actual values.

In an MLP, forward propagation first computes the predictions. A loss function then measures the difference between the actual and predicted values. Backpropagation computes the gradient of this loss function, moving from the output layer back toward the input layer. Here the chain rule of differentiation is used to find how much each weight contributes to the loss.

The computed gradients are used to update the weights in an optimization algorithm such as gradient descent or Adam. Training usually proceeds like this:

1. Compute predictions from the input variables.
2. Compare predictions with actual values to compute the loss.
3. Compute the gradient for each weight via backpropagation.
4. The optimization algorithm updates the weights.
5. Repeat this process to reduce the loss.

For example, if a customer who actually cancelled was given a predicted churn probability of 20%, the model records a large prediction error. The training algorithm adjusts the weights to reduce such errors.

However, the loss doesn’t necessarily decrease every time the weights are updated. Depending on the learning rate, the optimization method and how the mini-batches are formed, the loss can rise temporarily or fluctuate.

MLP classification

In a classification problem we predict which category an observation belongs to. For example, we can predict whether a customer cancels, whether a loan goes delinquent, or whether a product is defective.

Binary classification typically uses a sigmoid output with binary cross-entropy loss. For multiclass classification, a softmax output with categorical cross-entropy loss is common.

MLP regression

In a regression problem we predict a continuous value. For example, we can predict a customer’s purchase amount next month, house prices, production volume or test scores. A typical regression MLP uses a linear output layer and a loss function such as mean squared error (MSE).

So an MLP isn’t a method that analyzes only one particular type of outcome variable; it’s a model that can be applied to a variety of prediction problems by setting the output layer and loss function appropriately.

Worked example: predicting service cancellations

Suppose a subscription service company uses data on 5,000 customers to predict whether each will cancel next month.

Of all customers, 650 cancelled and the remaining 4,350 stayed. The overall churn rate is 13%. The analysis used the following variables.

The research question is:

Suppose the full data were split into 4,000 customers for training and 1,000 for final evaluation. Standardization and encoding of categorical variables were done within the training data, and hyperparameters such as the hidden-layer structure and the learning rate were chosen there. Predictive performance was then checked on the 1,000 customers held out for final evaluation.

All three models used the same training/evaluation split, and hyperparameters were chosen by cross-validation within the training data.

Predictive performance by model

These figures do not mean one algorithm is generally better than another.

In this example the MLP had the highest ROC-AUC at 0.86, and its PR-AUC was 0.51. This means that, on this evaluation data, the MLP produced a prediction ranking that was more useful than the two comparison models for separating customers who cancel.

Its log loss was also the lowest at 0.297. A lower log loss means relatively good results in terms of the predicted probability assigned to the actual category as well. But from this result alone we can’t conclude that the MLP is the best algorithm for every churn prediction problem.

MLP results can vary with the training settings, the initial weights, the size of the data and so on. So, where possible, it’s a good idea to also check how stable performance is by using several initial values or repeated evaluations.

Adding hidden layers and neurons makes the relationships the model can represent more complex. But it can also increase the amount of training data and computation needed and raise the risk of overfitting. For example, we might compare structures like these:

The numbers in parentheses are the number of neurons in each hidden layer.

Model A has a simple structure, so it may train fast and carry a relatively low risk of overfitting. Model D can represent more complex patterns, but with few observations it may turn into an unnecessarily complex model. The key point is that more hidden layers don’t automatically mean better predictive performance.

So it’s best to choose the hidden-layer structure based on performance in cross-validation or on validation data.

What is the learning rate?

The learning rate is the setting that decides how large a step to take when updating the weights.

For example, with a large learning rate the weights can change quickly, so the loss may drop fast early in training. But if the learning rate is too large, the weights may keep oscillating around the optimum or training may become unstable. Conversely, if the learning rate is too small, the weights change slowly, so training can take a long time.

The learning rate should be considered together with the optimization algorithm. For example, SGD and Adam adjust the weights differently, and the suitable range of learning rates can differ between them. So you shouldn’t assume that a particular learning rate suits every MLP.

What are epochs and batch size?

In an MLP you can set how many times training repeats and how many observations are processed at once.

Epoch

An epoch is one pass in which the entire training dataset is used for training. For example, if there are 4,000 training observations and you set 100 epochs, you have set up 100 training cycles over the full training data.

In actual implementations, though, the specific training process can differ depending on data shuffling, how the mini-batches are formed and so on.

Batch size

The batch size is the number of observations used for one weight update. For example, with 4,000 training observations and a batch size of 100, typical mini-batch training performs about 40 updates per epoch. A small batch size means more updates and can make the gradient estimates more variable.

Conversely, a large batch size processes more data at a time, which can increase memory use. Because epochs and batch size can affect training time, convergence and generalization performance, they should be reviewed together with the other settings.

For example, suppose tenure ranges from 1 to 60 months, the monthly fee is in the tens of dollars, and the number of customer-service inquiries runs from about 0 to 10. When input variables have such different ranges, weight optimization can proceed inefficiently or training can become unstable.

So in an MLP it is generally a good idea to standardize numeric input variables or scale them in a suitable way. A common method is z-score standardization.

For example, if tenure in the training data has a mean of 24 months and a standard deviation of 12 months, a customer with a tenure of 36 months has a standardized value of 1.

So split the data first, and apply the preprocessing parameters computed on the training data to the validation and evaluation data in the same way.

But if the model is too complex or there isn’t enough training data, it can overfit, learning even chance patterns in the training data. For example, suppose we got the following results.

Predictive performance across training epochs

The ROC-AUC on the training data kept rising as epochs increased. The ROC-AUC on the validation data, however, peaked at 60 epochs and fell after that. This suggests that the longer training went on, the worse generalization to new data may have become.

So with an MLP, don’t look only at performance on the training data; also watch how performance on the validation data changes.

What methods are used to reduce overfitting?

MLPs can use a variety of regularization methods to ease overfitting.

L2 regularization (weight decay)

L2 regularization keeps the weights from growing too large. It reduces the model’s heavy reliance on a few weights and encourages a simpler set of parameters.

Dropout

Dropout randomly leaves out the outputs of some neurons during training. By reducing over-reliance on particular neurons or connections, it can help ease overfitting. Not every MLP implementation supports dropout, though, so check what your library offers.

Early stopping

Early stopping halts training once performance on the validation data stops improving. For example, even if training is set to run up to 100 epochs, if validation performance keeps getting worse after 60 epochs, you can choose the model from an earlier point.

Limiting model complexity

Reducing the number of hidden layers and neurons to limit the model’s complexity is another approach. A model of appropriate size can perform better on new data than a neural network that is far too large for the amount of data.

These methods can be used together, but whether they actually improve performance should be checked on validation data.

For example, even if customers with short tenure get high predicted churn probabilities, it’s hard to read directly from any single hidden-layer weight how much tenure contributed to the prediction. So with MLPs, variable importance and prediction-explanation techniques can be used alongside.

Permutation importance

Permutation importance checks how much the model’s predictive performance drops when the values of a particular variable are randomly shuffled. For example, if shuffling contract type lowered PR-AUC from 0.51 to 0.41, that variable likely provided important information for the model’s predictions.

However, when variables are highly correlated with one another, individual importances can be hard to interpret.

SHAP

SHAP explains how much each variable contributed to a particular prediction. For example, if a particular customer’s predicted churn probability is high, you can look at how tenure, monthly fee, number of customer-service inquiries and so on pushed the prediction score up or down.

However, SHAP results can vary with the explanation method used, the background data, the output scale and so on.

Partial dependence (PDP)

A PDP shows how the model’s average prediction changes as the value of a particular variable is varied. For example, you can see how the average predicted churn probability changes as tenure gets longer.

But these results describe the predictive relationship the model has learned; they don’t carry the causal meaning that actually changing tenure would change the churn rate in the same way.

Depending on the characteristics of the data and the purpose of the analysis, it can be compared with logistic regression, decision trees, random forests, gradient boosting and others.

Logistic regression can be useful when you want to explain the relationship between variables and the outcome fairly clearly. A random forest can learn nonlinear relationships and interactions among variables while giving fairly stable predictions.

An MLP can represent more complex patterns through several layers of computation, but it is sensitive to its settings and can be affected by the size of the training data and the preprocessing. In particular, on typical tabular data, tree-based ensembles sometimes perform as well as or better than an MLP, so it’s important to compare actual performance.

Consider data size and model complexity together.

As hidden layers and neurons are added, the model’s capacity grows, but so does the number of weights to learn. Using an overly large model when there are few observations makes overfitting more likely. So choose a network structure that fits the size of the data.

Training results can depend on the initial values.

An MLP usually initializes its weights at random before training. Even with the same data and settings, the final model’s performance can differ slightly depending on the initial values or the order of the mini-batches. So it’s good practice to manage the random seed and, if needed, train repeatedly from several initial values to check that the results are stable.

Check for class imbalance.

With data where the category of interest is rare, as with customer churn, accuracy alone can’t tell you how well the model performs. For example, if 13% of customers cancel, predicting that every customer stays still gives 87% accuracy. So also check precision, recall, the F1 score, PR-AUC and so on.

Don’t take predicted probabilities at face value.

An output of 0.8 from an MLP’s output layer doesn’t guarantee that exactly 8 out of 10 similar customers will actually experience the event. When using predicted probabilities for real decisions, it’s a good idea to check their calibration on separate evaluation data.

Watch out for data leakage.

If standardization, missing-value imputation, encoding of categorical variables and so on are applied to the full data first, information from the evaluation data can leak into training. So split the data first and estimate the preprocessing parameters from the training data only. Also, the final evaluation data should not be used for hyperparameter tuning or early stopping.

In particular, because MLP results can vary with the number of hidden layers and neurons, the activation function, the optimization algorithm, the learning rate and so on, these settings should be reported clearly. Using the churn example from earlier, a report could read as follows.

In actual research, it’s good to also report the final hidden-layer structure, activation function, optimization algorithm, learning rate, number of epochs, batch size, regularization settings and so on. You can also check the gap between training and validation performance to assess the possibility of overfitting and, where needed, present the average performance or variability across repeated training runs.

Where can MLPs be used?

MLPs can be used for classification and regression problems that require learning complex relationships among many input variables.

Marketing and customer analytics

You can predict whether customers will cancel a service, how likely they are to buy, repurchase amounts, customer lifetime value and more.

Finance and economics

They can be used in studies predicting loan delinquency risk, the likelihood of default, anomalous financial transactions, continuous financial performance measures and more.

Health and medical research

You can use multiple test results and patient characteristics to predict particular health outcomes or risk levels. Applying this in actual clinical settings, however, requires thorough external validation and safety evaluation.

Education research

You can predict students’ academic achievement, likelihood of dropping out, completion of educational programs and more.

Manufacturing and quality control

They can be used to predict product defects, quality metrics, the likelihood of equipment failure, production volume and more.

Service operations and IT

They can be used to predict customer inquiry volume, service usage patterns, system anomalies and more. So although MLPs can be used in many fields, depending on the size and structure of the data a simpler statistical model or another machine learning algorithm may be more appropriate.

So when applying an MLP, weigh the actual prediction goal and validated performance more heavily than the complexity of the model itself.

Each neuron applies weights to its inputs and transforms the information through an activation function. During training, forward propagation computes the predictions, and backpropagation together with an optimization algorithm adjusts the weights in the direction that reduces the loss.

MLPs can be used for both classification and regression and are useful for learning from data with complex nonlinear relationships and interactions among many input variables.

But more hidden layers or neurons don’t always produce a better prediction model. Choose a model structure that fits the size of the data, and prevent overfitting with standardization, regularization, early stopping and the like.

Also, because an MLP’s predictive performance can vary with the initial weights and training settings, it’s best to compare it with other algorithms under the same evaluation conditions.