Logistic Regression

How likely is a customer to cancel a service? Can you predict whether a student passes an exam from study time and attendance? Can you analyze which of several products a customer will choose, or what factors are related to their level of satisfaction?

Logistic regression is the go-to statistical method for analyzing how likely an outcome is to fall into a particular category.

Where ordinary linear regression is used to explain continuous numbers such as test scores or sales, logistic regression is mainly used to analyze outcomes divided into categories, such as pass vs fail, purchase vs no purchase, or keeping vs canceling a service.

It is used in many fields, including customer churn prediction, medical risk factor analysis, surveys, education research, and marketing. In this article we cover the basic concepts of logistic regression, what probabilities and odds ratios mean, a worked example, the differences among binary, multinomial, and ordinal logistic regression, and points to watch out for when interpreting the results.

For example, suppose you collected the tenure, monthly fee, number of customer service contacts, and so on for 1,000 customers of a wireless carrier. Whether each customer kept or canceled the service is also recorded. With these data, you can see how customers’ usage characteristics are related to whether they cancel.

Based on these relationships, logistic regression can estimate the probability that a customer cancels the service. For example, if a particular customer’s predicted cancellation probability is 0.75, the model estimates that the probability of cancellation among cases with that customer’s characteristics is 75%.

This is, however, a probability calculated by a statistical model; it does not mean the customer will definitely cancel. To judge how reliable the predictions are, also check the model’s performance and how well its probabilities are calibrated.

How do linear regression and logistic regression differ?

Linear regression and logistic regression both analyze the relationship between independent and dependent variables, but they are typically used with different types of dependent variables.

For example, if you predict test scores from hours studied, you can use linear regression. If instead you analyze the probability of passing an exam from hours studied, you can consider logistic regression.

An ordinary linear regression equation calculates the predicted value of the dependent variable directly. But when the outcome is coded 0 or 1, such as pass/fail, a linear regression equation can predict values below 0 or above 1.

To solve this problem, logistic regression uses the logistic function so that predicted probabilities stay between 0 and 1.

The logit is the natural log of the odds, the ratio of the probability that the event occurs to the probability that it does not. The usual binary logistic regression equation is as follows.

Here p is the probability that the event of interest occurs. For example, if you analyze whether customers cancel a service, p is the probability of cancellation. The independent variables could include tenure, monthly fee, number of customer service contacts, and so on.

Logistic regression does not assume a straight-line relationship between the independent variables and the probability itself; it sets up a linear relationship between the independent variables and the log odds. This structure lets you estimate the probability of the event and interpret the relationship between the independent variables and the outcome through odds ratios and similar measures.

How do probability and odds differ?

To understand logistic regression, you need to know the difference between probability and odds.

Probability expresses how likely an event is as a value between 0 and 1. Odds, on the other hand, are the probability that the event occurs divided by the probability that it does not.

For example, suppose the probability that a customer cancels the service is 20%. Then the probability of not canceling is 80%, and the odds are as follows. 0.20 ÷ 0.80 = 0.25

Conversely, if the probability of cancellation is 75%, the odds are as follows. 0.75 ÷ 0.25 = 3.00

When the probability is 50%, the odds are 1, because the event is equally likely to happen or not happen. When the probability is above 50%, the odds are greater than 1, and when it is below 50%, the odds are less than 1. Logistic regression estimates relationships between variables using the log of these odds, the log odds.

The dependent variable is whether the service was canceled, coded 1 for customers who canceled and 0 for customers who kept the service. The independent variables were set up as follows.

• Tenure: number of months the customer has used the service
• Monthly fee: the amount paid each month
• Customer service contacts: number of contacts in a recent period

The research question is as follows.

Logistic regression results

How do you interpret the results?

The odds ratio for tenure was 0.960. Holding the other variables constant, each additional month of tenure is associated with about 4.0% lower odds of cancellation.

The odds ratio for monthly fee is 1.200. You can interpret this as a $20 increase in the monthly fee being associated with about 20.0% higher odds of cancellation. The odds ratio for customer service contacts is about 1.499, meaning each additional contact is associated with about 49.9% higher odds of cancellation.

In this example, all three variables were statistically significant.

The odds ratio shows by what factor the odds of the event change when an independent variable changes by one unit. In a model that includes other independent variables, it is interpreted with those variables held constant. The odds ratio is calculated by applying the exponential function to the regression coefficient.

The usual interpretation is as follows.

For example, OR = 1.50 means that when the independent variable increases by one unit, the odds of the event become 1.5 times as large. OR = 0.70, on the other hand, means the odds become 0.7 times as large, which corresponds to 30% lower odds.

Odds ratios are useful for understanding the direction and relative size of a relationship, but you must always check the units of the independent variable when interpreting them. For example, the odds ratio for monthly fee is different when the fee is measured per $1 than when it is measured per $20.

How much does the actual probability change when odds increase?

People often confuse odds ratios with probabilities in logistic regression.

For example, suppose a variable has an odds ratio of 2. This means the odds of the event double when that variable increases by one unit. But it does not mean the probability always doubles. The actual change in probability depends on the baseline probability.

For example, for a customer whose baseline cancellation probability is 10%, doubling the odds only brings the cancellation probability to about 18.2%. If the baseline cancellation probability is 50%, doubling the odds brings it to about 66.7%. So even with the same odds ratio, the amount of change in actual probability can differ depending on the baseline probability.

In practice, results are easier to understand if you present not only odds ratios but also predicted probabilities calculated under specific conditions, average marginal effects, and so on.

The main types are binary logistic regression, multinomial logistic regression, and ordinal logistic regression.

All three methods analyze categorical outcomes, but because the structure of the dependent variable differs, you need to choose the appropriate one.

Typical examples are as follows.

• Cancel / keep a service
• Buy / don’t buy a product
• Pass / fail an exam
• Disease occurs / does not occur
• Click / don’t click an email

For example, if you analyze whether customers cancel a service, you can code cancellation as 1 and keeping the service as 0. Binary logistic regression estimates the relationship between the independent variables and the log odds of the event. Based on this, you can calculate each customer’s probability of cancellation or analyze how strongly particular factors are related to the odds of cancellation.

However, the direction in which coefficients and odds ratios are interpreted depends on which category was coded 1. So when reporting results, explain clearly how the event of interest was defined.

For example, suppose you surveyed the means of transportation people mainly use to commute. The response options are as follows.

• Car
• Public transit
• Bicycle

These three categories have no natural order such as higher or lower. Multinomial logistic regression usually sets one category as the reference category and analyzes how the other categories compare with it.

For example, if car is set as the reference category, you can estimate the following relationships.

• The relative likelihood of choosing public transit vs choosing a car
• The relative likelihood of choosing a bicycle vs choosing a car

The independent variables can include age, income, commuting distance, whether the person owns a car, and so on. In multinomial logistic regression, results are sometimes expressed with relative risk ratios (RRR) relative to the reference category.

However, the relative risk ratio here should not be interpreted the same way as the risk ratio in a typical cohort study. In a multinomial logit, it shows how the ratio of the probability of a particular category to that of the reference category changes with the predictors.

For example, suppose service satisfaction was measured as follows.

1. Very dissatisfied
2. Dissatisfied
3. Neutral
4. Satisfied
5. Very satisfied

In data like these, the order of the values is meaningful. But you cannot necessarily assume that the difference between 1 and 2 points is the same as the difference between 4 and 5 points. Ordinal logistic regression takes this order information into account when analyzing the relationship between the independent variables and the level of satisfaction.

One of the main methods is the proportional odds model. This model uses the proportional odds assumption, that the effect of an independent variable on the cumulative category probabilities is the same across category boundaries. For example, you can check whether higher service quality increases the likelihood of falling into a higher satisfaction category.

If the proportional odds assumption is not appropriate, however, you may need to consider a partial proportional odds model or another ordinal regression model.

How do you tell the three types of logistic regression apart?

The first thing to check when choosing a logistic regression method is the measurement level of the dependent variable.

For example, if satisfaction scores range from 1 to 5, it may not be appropriate to use multinomial logistic regression just because there are five categories. Satisfaction is an ordered response, so ordinal logistic regression should be considered first.

Conversely, if preferred brands are divided into A, B, and C, there is no order among the categories, so you can consider multinomial logistic regression. In this way, the method depends not only on the number of categories but also on whether there is a meaningful order among them.

Ordinary linear regression expresses the conditional mean of a continuous dependent variable as a linear equation. A GLM, by contrast, can use different probability distributions and link functions depending on the type of outcome being analyzed. Typical examples are as follows.

Binary logistic regression is a classic example of a GLM that uses the binomial distribution and the logit link function. Poisson regression is also a type of GLM, used to analyze counts of events over a period of time or per unit of exposure.

However, not every logistic regression model has exactly the same structure as a basic GLM. Multinomial and ordinal logits are often handled within the framework of generalized categorical response models.

For binary classification problems in particular, you can use the following measures.

For example, suppose 90% of customers keep the service and only 10% cancel. Predicting that every customer will keep the service gives 90% accuracy. But this model does not find a single customer who actually cancels.

So with data where the event of interest is rare, like cancellations, it is better to look at precision, recall, PR-AUC, and so on rather than accuracy alone. Calibration, which checks how well predicted probabilities match the actual event rates, also matters.

Type of dependent variable

In binary logistic regression, the dependent variable must fall into two categories. Multinomial logistic regression handles several unordered categories, and ordinal logistic regression handles several ordered categories. So check the measurement level of the dependent variable carefully before the analysis.

Independence of observations

Ordinary logistic regression models treat independence between observations as an important condition. For example, if you treat data from the same customer at several time points as if they came from independent customers, standard errors and test results can be inaccurate. In such cases you can consider mixed-effects logistic regression or generalized estimating equations (GEE).

Linear relationship with the log odds

Binary logistic regression considers linearity between continuous independent variables and the log odds, not the probability of the event itself. For example, if there is a complex nonlinear relationship between tenure and the likelihood of cancellation, a model with only a simple linear term may not represent the relationship adequately.

If needed, you can use polynomial terms, splines, and so on.

Multicollinearity

If the relationships among independent variables are too strong, estimates of individual coefficients can become unstable. For example, including very closely related variables together, such as monthly fee and annual spending, can cause multicollinearity.

Complete separation

When values or combinations of a particular independent variable perfectly separate the outcome categories, ordinary maximum likelihood estimation may produce coefficients that diverge or cannot be estimated stably. In such situations, consider other estimation methods such as Firth’s correction or regularized regression.

Enough events

Even if the total sample is large, it can be hard to estimate coefficients stably when the event of interest is very rare. For example, if only 20 of 10,000 customers cancel, a model with many independent variables can overfit or produce unstable estimates.

The required sample size and number of events should be decided by considering the number of predictors, the expected effects, the event rate, the research goal, and so on.

Don’t confuse odds ratios with probabilities.

An odds ratio of 1.5 does not mean the probability of the event increases by 50 percentage points. It means the odds become 1.5 times as large, other things being equal. The actual change in probability depends on the baseline probability.

Statistical significance does not guarantee predictive performance.

A small p-value for a particular independent variable does not mean the model can accurately predict whether new customers will cancel. If prediction is the goal, check the model’s performance with separate validation data or cross-validation.

Regression coefficients do not imply causation.

Just because higher monthly fees were associated with higher odds of cancellation, you cannot conclude that lowering fees will necessarily reduce the cancellation rate. Other factors, such as customer characteristics, contract type, and service experience, may be at work as well.

Check model fit as well.

Logistic regression offers various measures such as the log-likelihood, AIC, BIC, and pseudo R². Pseudo R², however, is calculated and interpreted differently from the R² of ordinary linear regression, so it should not be interpreted the same way.

Using the customer cancellation example above, you could write it as follows.

In real studies, it is good to also report the 95% confidence interval of each odds ratio, model fit, the analysis sample size, and so on. And if the goal is to predict the probability of the event, you can also present predictive performance and calibration results.

Marketing and customer analytics

You can analyze whether customers make a purchase, cancel a subscription, click an ad, buy again, and so on.

Education research

You can predict, or analyze the factors related to, whether students pass, drop out, complete an education program, and so on.

Health and medical research

You can analyze factors related to binary outcomes such as whether a disease occurs, whether there is a response to treatment, and whether a patient is readmitted.

Social science research

You can analyze variables related to categorical outcomes such as employment, support for or opposition to a policy, vote choice, and level of satisfaction.

Finance and economics

You can analyze a variety of categorical outcomes, such as loan delinquency, the likelihood of default, and customers’ choice of financial products.

HR and organizational research

You can analyze whether employees leave, whether they take part in training programs, their level of job satisfaction, and so on. In this way, logistic regression is widely used to estimate the probability of a particular event or to analyze the relationship between a categorical outcome and several predictors.

The main types are binary, multinomial, and ordinal logistic regression, and you should choose the appropriate one based on how many categories the dependent variable has and whether they are ordered.

When reading the results, it is important to look not only at coefficients and p-values but also at odds ratios and confidence intervals. In particular, odds ratios and probabilities are different concepts, so take care when interpreting them. The measures you need to check also depend on whether the research goal is to explain relationships between variables or to predict outcomes for new cases.