Life Table Analysis

A company that runs a subscription service wants to know how long customers keep the service after signing up. How many customers cancel in the first 3 months? Are customers who make it past 6 months more likely to keep the service after that? Do cancellations cluster in particular periods?

To answer these questions, a simple overall cancellation rate isn't enough. You need to divide the time customers have used the service into periods and look at how many customers canceled in each one. A method you can use for this is life table analysis.

Life table analysis is a survival analysis method that divides the observation period into fixed intervals and calculates the probability of the event and the survival probability for each interval. You can see the proportion that remains without the event as time goes on, and also check whether the risk of the event rises in particular intervals.

It can be used to analyze patients' survival times in medical research and the time until customers cancel a service in customer analytics. It also applies to many other studies where the time until an event matters, such as when a machine breaks down, when an employee leaves, or when an unemployed person finds a new job.

In this article we'll cover the basic concepts of life table analysis, calculating interval survival probabilities, handling censoring, cumulative survival probabilities and hazard rates, how it differs from the Kaplan–Meier estimator, and a real analysis example with interpretation of the results.

In survival analysis, "survival" doesn't necessarily mean a person staying alive. It means the state in which the event defined by the analyst has not yet occurred.

For example, if the event in a customer churn analysis is defined as canceling the service, the survival probability is the probability that a customer keeps the service without canceling. If the event in manufacturing is defined as a machine failure, the survival probability is the probability that the machine keeps running without failing.

In life table analysis, the observation period is divided into intervals such as 0–3 months, 3–6 months, and 6–9 months. Then the number of subjects entering each interval, the number of events, and the number censored are used to calculate the survival probability for each period. Accumulating these interval results lets you estimate the survival probability over the whole observation period.

What sets life table analysis apart is that it doesn't calculate a probability at the exact time of each individual event; instead, it estimates survival probabilities based on time intervals set in advance.

For example, suppose a company tracks how long customers have used its service in 3-month units rather than exact days. In that case, rather than analyzing cancellation times by individual dates, you can divide time since sign-up into intervals such as 0–3 months, 3–6 months, and 6–9 months and look at the risk of cancellation.

In medical research, too, if patients' status is checked at scheduled visits, you can summarize the survival pattern by period. But if the data are interval-censored, with the actual event time known only to fall between two visits, you may need a method that reflects that structure directly rather than a simple life table analysis.

Typical situations where life table analysis fits well include:

Life table analysis is good for summarizing the pattern of events by period in a single table. But if event times are recorded exactly, the Kaplan–Meier estimator can make more detailed use of the time information.

Where can you use life table analysis?

Life table analysis is used in many fields where you need to analyze the time until a particular event by interval.

Medical and health research

You can analyze, by interval, patients' survival time after treatment, the time until disease recurrence, or the time until a particular health outcome occurs.

Marketing and customer analytics

You can analyze the time from sign-up to cancellation, customer retention rates, the churn risk of a subscription service by period, and so on.

HR and organizational research

You can analyze the time from hiring to resignation, the time to promotion to a particular level, the time until dropping out of a training program, and so on.

Manufacturing and quality control

You can analyze the risk of machine or equipment failure by length of use, product lifetimes, the time until a part is replaced, and so on.

Labor economics and social science research

You can analyze the time from job loss to reemployment, the length of job searches, or the time until a particular social event occurs.

Education research

You can analyze the time from enrollment to dropping out, the time to complete a degree, the time to finish an education program, and so on. As these examples show, life table analysis is useful in studies that focus on understanding the pattern of events over set periods rather than analyzing exact individual event times.

For example, to analyze the time until customers cancel a service, you need to know the sign-up date and the cancellation date or last observation date. If a customer did not cancel during the observation period, that customer can be treated as a censored observation.

Example data layout

The analysis uses a time variable for the observation time and a status variable for whether the event occurred. The status variable is usually coded 1 for an event and 0 for censoring, but how it's set up can vary by software.

The researcher also has to decide on the time intervals to analyze. For example, you can set intervals of one month, 3 months, 6 months, and so on. Because the length of the intervals affects the results, it's important to choose it to suit the characteristics of the data and the research aim.

For example, suppose a customer used the service for 6 months and had not canceled by the end of the study. You can't know when this customer will cancel in the future. But you do know that they kept the service for at least the 6 months they were observed.

An observation like this is called right-censored. Treating censored customers as cancellations can overestimate the risk of cancellation. On the other hand, dropping them all from the analysis throws away the information that they kept the service during the observation period.

Life table analysis adjusts the number at risk in each interval to account for censoring. The standard life table calculation assumes censoring occurs fairly evenly within an interval and subtracts half of the interval's censored cases from the number at risk.

This is, however, a computational approximation. If censoring is concentrated at particular times or is related to the likelihood of the event, the estimates can be biased.

The main components are:

Laying out these results by time interval shows not only the overall survival probability but also which periods had many events. For example, if the event probability is high in the first 3 months after sign-up, you might focus on managing new customers.

Conversely, if you find an interval where the event probability rises among long-term customers, you can analyze the characteristics of that period further. But a high event probability in an interval alone doesn't let you conclude that some particular cause was at work in that period.

How do you calculate the effective number at risk?

In life table analysis, you can calculate the effective number at risk for each interval, taking censoring into account.

The standard actuarial life table uses the following approximation.

For example, suppose 100 customers were under observation at the start of an interval. If 10 of them canceled and 20 were censored during the interval, the effective number at risk is: 100 - (20 ÷ 2) = 90

Since 10 customers had the event in this interval, the event probability is estimated as: 10 ÷ 90 ≈ 0.111 So the interval event probability is about 11.1%, and the interval survival probability is about 88.9%.

This calculation approximates that censored customers contributed to the number at risk for about half of the interval on average. But if the actual censoring times are recorded exactly, you can use the Kaplan–Meier estimator, which reflects event times and changes in the number at risk in more detail.

The observation period is 12 months after sign-up, and a cancellation is recorded as an event. Customers who still had the service at the end of the study, or whose observation stopped, are treated as censored. For the analysis, the observation period was divided into 3-month intervals.

Observed results by interval

In the first interval, 120 of the 1,000 customers canceled and 40 were censored. So 840 customers remain under observation at the start of the next interval. In the 3–6 month interval, 80 canceled and 60 were censored, leaving 700 at 6 months.

In this way, you can use the number of events and the number censored in each interval to calculate interval survival probabilities.

How do you calculate interval survival probabilities?

Let's calculate survival probabilities from the data on 1,000 customers presented above.

First interval: 0–3 months

The number at the start of the interval is 1,000 and the number censored is 40. The effective number at risk is: 1,000 - (40 ÷ 2) = 980

There were 120 cancellations, so the interval event probability is: 120 ÷ 980 ≈ 0.1224 So the interval survival probability is:

1 - 0.1224 ≈ 0.8776 That is, the conditional probability of keeping the service during the first 3 months is estimated at about 87.8%.

Second interval: 3–6 months

The number at the start of the interval is 840 and the number censored is 60. The effective number at risk is: 840 - (60 ÷ 2) = 810

There were 80 cancellations, so the interval event probability is: 80 ÷ 810 ≈ 0.0988 So the interval survival probability is about 90.1%.

This is the conditional probability that customers who kept the service up to the start of the interval at 3 months, and remained under observation, continue to keep it during that interval. To get the cumulative survival probability, multiply the survival probability of the first interval by that of the second interval. 0.8776 × 0.9012 ≈ 0.791

So the cumulative probability of keeping the service through 6 months after sign-up is estimated at about 79.1%.

The formula is:

Using the data above gives the following results.

Life table results

Values calculated with the actuarial life table method; displayed values are rounded.

In these results, the cumulative probability of keeping the service through 3 months after sign-up is about 87.8%. It falls to about 79.1% by 6 months, about 73.2% by 9 months, and about 68.1% by 12 months. In other words, in this example the probability of not canceling within 12 months of sign-up is estimated at about 68.1%.

This is not the same, however, as simply dividing the number of customers remaining at 12 months by the initial number of customers, because life table analysis takes censored customers into account when calculating the interval event probabilities and cumulative survival probabilities.

How do you visualize life table results?

In life table analysis, you can plot the cumulative survival probability over time.

The horizontal axis shows the observation time and the vertical axis the cumulative survival probability. In the customer retention example above, the results were:

Plotting these results shows how the probability of keeping the service decreases over time. Because the survival probability is the probability of remaining event-free, a typical survival curve never increases over time.

In particular, a large drop in survival probability in a certain interval suggests that relatively many events may have occurred in that period. But because the number under observation differs by interval and censoring can occur, you need to look at the interval event probabilities, not just the raw number of events.

The example above gave the following results.

In these results, the highest interval cancellation probability was in months 0–3 after sign-up, and the cancellation probability gradually decreased in the later intervals. So in this example you can say that the risk of churn was observed to be relatively high among new customers.

Here, though, you need to distinguish the interval event probability from the hazard rate. The interval event probability is the conditional probability of the event occurring during the period. The hazard rate, in contrast, is the intensity of the event at a given time, given that the subject has remained event-free up to that time.

In a life table you can approximate the hazard over an interval from the interval event probabilities, but the two measures should not be used to mean the same thing.

How do you find the median survival time?

The median survival time is the time at which the cumulative survival probability first falls to 50% or below.

For example, suppose the cumulative survival probabilities looked like this.

In these results, the cumulative survival probability is 58% at 9 months and 45% at 12 months. So the median survival time lies in the 9–12 month interval. Because grouped life tables often don't know the exact event times, interpolation within the interval can be used to estimate the median survival time.

So the estimate of the median survival time can differ slightly depending on the calculation method. In the 1,000-customer example calculated earlier, on the other hand, the cumulative survival probability at 12 months is about 68.1%.

In other words, the survival probability never fell to 50% or below during the observation period, so the median retention time was not reached. In this case, you should not arbitrarily report the median survival time as 12 months.

For example, suppose you compare how long customers on monthly contracts and annual contracts keep the service. Applying the same time intervals to both groups lets you compare the probability of keeping the service by period.

Cumulative survival probability by contract type

In this example, customers on annual contracts had a higher probability of keeping the service at every time point. In particular, the cumulative retention probability at 12 months was 54% for monthly contract customers and 85% for annual contract customers.

So you can say that annual contract customers tended to keep the service relatively longer during the analysis period. But the fact that the two groups' survival probabilities differ doesn't by itself let you conclude that there is a statistically significant difference.

To test the difference in survival experience between groups statistically, you can additionally use a method such as the log-rank test.

The biggest difference between them is how they handle time. The Kaplan–Meier estimator calculates the survival probability at each individual event time. Life table analysis, in contrast, divides the observation period into several intervals and calculates the survival probability in each.

For example, if customers' cancellation dates are recorded exactly, you can use the Kaplan–Meier estimator to calculate the survival probability at every event time. If cancellation records are summarized in 3-month units instead, life table analysis can be useful.

That said, even if the data include exact event dates, you can also run a life table analysis if the research aim is to compare periods. So the two methods aren't complete substitutes for each other; rather, you choose between them according to the form of the time data and the aim of the analysis.

How do life table analysis and Cox regression differ?

Life table analysis is mainly used to estimate survival probabilities by time interval.

Cox proportional hazards regression, on the other hand, analyzes the relationship with the risk of the event while considering several explanatory variables at once. For example, suppose you compare how long customers on monthly and annual contracts keep the service.

Life table analysis shows the cumulative survival probabilities and interval event probabilities by contract type. But if you want to consider other variables at the same time, such as how long the customer has used the service, the monthly fee, and how often they use it, Cox regression may be more appropriate.

So life table analysis is useful for looking at the pattern of events by period, and Cox regression is useful for analyzing the relationship between a particular variable and event risk while accounting for other factors.

Censoring must be non-informative.

If censoring is closely related to the likelihood of the event, the survival probabilities can be biased. For example, if customers more likely to cancel also drop out of the study more often, survival probabilities calculated from the remaining customers' data can come out higher than they really are.

Be careful with how censoring within intervals is handled.

The standard actuarial life table assumes that censored subjects contributed to observation for about half of the interval on average. But if censoring is concentrated at particular times, this approximation may not be appropriate.

Time intervals must be set appropriately.

If the intervals are too wide, it's hard to see detailed differences in event timing. If they are too narrow, some intervals may have hardly any events or a very small number at risk. So choose the time intervals with the sample size, event frequency, research aim, and so on in mind.

Consider the independence of observations.

Standard life table analysis assumes that the event times of the subjects are independent of one another. Treating repeated contracts by the same customer or multiple events in the same patient as independent observations can distort the results. So if there are recurrent events or a clustered structure, consider an analysis method that accounts for it.

Why do confidence intervals matter in life table analysis?

Survival probabilities calculated by life table analysis are estimates based on sample data.

So there's no guarantee they exactly match the survival probabilities in the actual population. A confidence interval can be used to show the uncertainty of these estimates. For example, suppose the 12-month cumulative survival probability is 68.1% with a 95% confidence interval of 64.0% to 72.0%.

This expresses, as an interval, the uncertainty in the survival probability estimate under the statistical method and assumptions used. In particular, the longer the observation period, the more the number of subjects remaining at risk can shrink because of events and censoring. So survival probabilities later in follow-up are likely to have more estimation uncertainty than those at early time points.

When interpreting life table results, it's a good idea to look not only at the cumulative survival probabilities but also at the number at risk in each interval and the confidence intervals.

Results can depend on the interval length.

Even with the same data, dividing time into 1-month versus 6-month intervals changes how event probabilities are expressed and how censoring is approximated. So choose the intervals to suit the research aim, and where needed check whether the results change much under different interval settings.

Don't treat censoring as an event.

Subjects who had not experienced the event by the end of observation did not have the event. So miscoding censoring as an event can make the estimated survival probability lower than it really is.

Don't confuse interval event probabilities with hazard rates.

The interval event probability is the probability of the event occurring during the period. The hazard rate, in contrast, is the intensity of the event at a particular time. The two measures are related but are not the same concept.

Don't confuse survival probability with a simple proportion remaining.

The cumulative survival probability in a life table is an estimate that accounts for censoring. So it can differ from the number of subjects observed to the end divided by the initial number of subjects.

Don't interpret differences between groups as causal effects.

A difference in survival probabilities by contract type doesn't mean you can conclude that the contract type itself directly causes cancellation. Other factors, such as the customer's purpose of use, price, and usage habits, may also be involved.

Check for competing events.

If another event can occur before the event of interest and make it impossible, you need to consider competing risks. In that situation, reading the complement of an ordinary survival probability as the cumulative incidence of the event of interest can overestimate the actual risk.

Using the customer retention example above, you could write it up as follows.

In real research, it's best to also present the number at the start of each interval, the number of events, the number censored, the effective number at risk, and so on. Stating how censoring was handled and how confidence intervals were calculated also makes the results more reproducible.

If you compared life tables between groups, report each group's cumulative survival probabilities and, where needed, add results such as a log-rank test.

It considers not only whether the event occurred but also the time until it occurred, and it can include censored observations in the analysis.

Life table results show the interval event probabilities, interval survival probabilities, cumulative survival probabilities, median survival time, and so on. These let you see which periods events cluster in and how the survival probability changes over time.

Unlike the Kaplan–Meier estimator, life table analysis calculates based on time intervals set in advance. So for data in which individual event times are recorded exactly, the Kaplan–Meier estimator may be better suited to a more detailed analysis.

Also, because the interval length and the handling of censoring can affect the estimates, you need appropriate settings and checks of the assumptions.