Log-Rank Test

Suppose a hospital wants to compare how long patients survive after receiving one of two treatments. A Kaplan–Meier survival analysis shows that patients on Treatment A had a higher survival probability overall than patients on Treatment B. Can we conclude, then, that Treatment A led to better outcomes?

Even if the survival curves look different on the graph, the difference may have come from chance variation in the sample. So we need a procedure to check whether the two groups’ survival experience is statistically different. The standard method for this is the log-rank test.

The log-rank test is a nonparametric method for testing whether the time until an event differs between two or more groups. Besides patient survival times, it can be used with all kinds of survival data, such as how long customers keep a service, how long employees stay before leaving, or how long a machine runs before it breaks down.

In this lesson we cover the basic idea of the log-rank test, how it is actually calculated, how to interpret the test statistic and p-value, the problem of crossing survival curves, and how it differs from Kaplan–Meier estimation and Cox regression.

A survival function gives the probability that the event of interest has not happened by a given time. For example, in a customer churn analysis where the event is defined as canceling the service, the survival function is the probability that a customer is still with the service at a given time.

At each time an event occurs, the log-rank test compares the number of events actually observed in each group with the number expected if the two groups had the same event experience. If the gap between observed and expected events is large enough, we can reject the null hypothesis that the groups have the same survival experience.

However, the log-rank test does not directly estimate how many months apart the survival times are, or how many times higher one group’s risk of the event is than another’s. So alongside the test result, it is a good idea to look at the Kaplan–Meier survival curves, the survival probabilities at specific times, the median survival time, and so on.

When is the log-rank test used?

The log-rank test is used to compare the time until an event across several groups.

The key point is that it does not just compare the proportion of subjects who had the event; it also takes into account when the events happened and how long each subject was followed. For example, suppose we compare two treatments in 200 patients.

If 30 patients died in the Treatment A group and 40 died in the Treatment B group, the raw death counts might make Treatment B look worse. But if the two groups were followed for different lengths of time or had different numbers of censored patients, the raw death counts alone make it hard to compare their survival experience.

The log-rank test uses this time information to assess the difference in survival functions between groups.

The log-rank test is widely used in studies that want to check whether the distribution of survival times differs between groups.

For example, if we compare how long customers on monthly and annual contracts keep the service, the null hypothesis is that the two groups have the same survival function. The alternative hypothesis is that their survival functions differ. If the test’s p-value is smaller than the significance level set in advance, we can reject the null hypothesis.

For example, with a significance level of 0.05 and a p-value of 0.008, we would say there is statistical evidence against the hypothesis that the two groups have the same survival function. If instead the p-value were 0.20, we would say there is not enough evidence to reject the null hypothesis.

The important point here is that a p-value above 0.05 does not prove that the two groups have the same survival function.

Kaplan–Meier estimation calculates survival probabilities over time. The log-rank test, on the other hand, statistically tests whether different groups have the same survival function.

For example, a Kaplan–Meier analysis might show a 12-month survival probability of 80% in the Treatment A group and 65% in the Treatment B group. A log-rank test can then assess whether the two groups’ overall survival experience is statistically different.

So the two methods are not competitors; they complement each other.

Customers were split into two groups by contract type.

• Monthly contract: 200 customers
• Annual contract: 200 customers

The researcher recorded the time from sign-up until each customer canceled the service. The event of interest is cancellation, and customers who still had the service at the end of the observation period were treated as censored. Suppose the Kaplan–Meier survival probabilities for the two groups came out as follows.

Service retention probability by contract type

At every time point, customers on annual contracts had a higher retention probability. At 12 months, for example, the survival probability was 62% for monthly-contract customers and 86% for annual-contract customers. These numbers alone, however, do not let us conclude that the two groups’ survival functions differ significantly.

So we use the log-rank test to assess the difference in survival experience between the groups.

For example, suppose there are 100 people at risk in total, 60 in group A and 40 in group B. If 10 events occur at that time, then under the null hypothesis that the two groups have the same event experience, the expected number of events in group A is: 10 × (60 ÷ 100) = 6 events

The expected number of events in group B is: 10 × (40 ÷ 100) = 4 events But if the actual number of events was 3 in group A and 7 in group B, there is a gap from the expected counts.

The heart of the log-rank test is combining these gaps across all the event times. Rather than just comparing the total number of events, it also takes into account how many people are still at risk at each event time.

How is the expected number of events calculated?

In the log-rank test, the expected number of events in each group is calculated at each event time.

The expected number of events in group A is:

where the symbols mean:

• Eₐᵢ: expected number of events in group A at event time i
• dᵢ: total number of events at that time
• nₐᵢ: number at risk in group A just before that time
• nᵢ: total number at risk just before that time

For example, suppose that at a particular event time there are 80 people at risk in total, 50 in group A and 30 in group B. If 4 events occur in total at that time, the expected number of events in group A is: 4 × (50 ÷ 80) = 2.5 events

The expected number of events in group B is 1.5.

If 1 event actually occurred in group A and 3 in group B, then group A had fewer events than expected. The log-rank test calculates this kind of gap at each time point and adds them up. A single time point, though, is not enough to judge the overall difference in survival experience between the groups.

This total can be standardized by an appropriate variance to give the test statistic. The standard formula is:

Here O is the total number of observed events in one group, E is the sum of the expected events under the null hypothesis, and V is the variance of the difference between the two. Under suitable conditions this test statistic can be evaluated against a chi-square distribution with 1 degree of freedom. For example, suppose the calculation gave the following values.

These values were not calculated directly from the raw data of the 400 customers above.

The chi-square statistic is calculated as follows. χ² = (30 - 42)² / 24 = 6.00 Using a chi-square distribution with 1 degree of freedom, the p-value is about 0.014.

If the significance level is set at 0.05, we can reject the null hypothesis that the two groups have the same survival function. This result, however, only provides evidence that there is a statistical difference between the groups; it does not tell us how many months the difference is or which variables caused it.

For example, suppose the comparison of the two contract types gave the following result.

This result provides statistical evidence against the null hypothesis that the two groups have the same survival function. At a significance level of 0.05, we can say there is a statistically significant difference in survival experience between the two groups. But a p-value of 0.002 does not mean there is a 0.2% probability that the two groups have the same survival function.

Nor does a small p-value mean the difference is necessarily large or important in practice. So along with the p-value, you should look at the survival curves, the survival probabilities at each time point, the number of observed events, and so on.

For example, if the survival curve for annual-contract customers sits above the curve for monthly-contract customers throughout the observation period, we can say that annual-contract customers were observed to have a higher retention probability overall. Checking where the curves sit is not enough, though.

In particular, it helps to look at the following points too. First, check when the gap between the two curves starts to appear. Whether the difference is there from the start of the subscription or grows after a certain period can change what the result means.

Second, check the number at risk late in the observation period. As fewer subjects remain under observation over time, the survival probability estimates can become more uncertain. Third, check the confidence intervals. You should not judge statistical significance from the visual gap between survival curves alone.

Fourth, check whether the survival curves cross. If the two curves cross, extra care is needed when interpreting the log-rank test result.

What goes wrong when survival curves cross?

The log-rank test often has high power when the difference in event risk between the two groups stays in a fairly consistent direction throughout the observation period.

For example, if the Treatment A group has a lower risk of the event than Treatment B throughout the whole observation period, the log-rank test can be useful. But problems can arise when the two groups’ survival curves cross.

For example, consider a situation where Treatment A has a higher survival probability at first, but after some time Treatment B’s survival probability becomes higher.

In this case, the early and late differences in events can go in opposite directions. Because the log-rank test adds up the gaps between observed and expected events at each event time, these differences can partly cancel each other out.

As a result, even if the survival curves show a meaningful difference, the log-rank test may fail to give a significant result. So when survival curves cross, don’t just look at the log-rank p-value; also look at when the differences occur and in which direction.

If needed, you can consider comparing restricted mean survival time (RMST) or using analysis methods that allow for effects that change over time.

Depending on the research goal, however, you may want to give more weight to differences at certain times. For example, the difference in survival in the first 6 months after treatment starts may matter most, or you may be interested in long-term treatment effects.

In such cases you can consider a weighted log-rank test. A weighted log-rank test assesses the difference between groups by giving different weights to different event times. Common methods include the following.

In a weighted log-rank test, how sensitive the test is to differences at particular times depends on the weights used. So rather than choosing a test after seeing the results in order to get a significant p-value, it is best to decide in advance, based on the research goal and the expected shape of the survival curves.

For example, suppose we split customers by contract type into monthly, 6-month, and annual contracts and compare how long they keep the service. The survival probabilities of the three groups might look like this.

In this case, the overall log-rank test evaluates the null hypothesis that all three groups have the same survival function. Under suitable conditions, the test statistic of the usual K-group log-rank test follows a chi-square distribution with K-1 degrees of freedom. For example, suppose the log-rank test for the three groups gave the following result.

This provides statistical evidence against the hypothesis that all three groups have the same survival function. But this result alone does not tell us which pairs of groups differ. So if needed, we can look at each group’s survival curve and run post hoc pairwise comparisons.

How do you run a post hoc test?

If a log-rank test of three or more groups shows a significant overall difference, you can go on to check which groups differ from each other.

For example, when comparing customers on monthly, 6-month, and annual contracts, the following three pairwise comparisons are possible.

• Monthly vs. 6-month
• Monthly vs. annual
• 6-month vs. annual

A log-rank test can be applied to each comparison. But running many tests increases the chance of getting a significant result by chance. So you should consider a multiple comparison adjustment.

Common choices include the Bonferroni method and the Holm method.

Pairwise comparison example

In this example, at a significance level of 0.05, the differences between monthly and annual and between monthly and 6-month contracts were statistically significant. On the other hand, no significant difference was found between 6-month and annual contracts. A non-significant result, however, is not evidence that the two groups have the same survival function.

The log-rank test mainly tests whether there is a difference in survival functions between groups. Cox regression, on the other hand, estimates the relationship between explanatory variables and the risk of the event, and can take other variables into account at the same time.

For example, to check whether Treatments A and B lead to different survival experience, you can use the log-rank test. But if you want to look at the relationship between treatment and event risk while also accounting for patients’ age, disease severity, and so on, you can use Cox regression.

In particular, a significant difference in the log-rank test does not guarantee a significant result in a Cox regression that adjusts for other factors. So choose between the two methods based on the purpose of the analysis.

Censoring should be non-informative.

Under suitable conditions, the event process of censored subjects should be comparable to that of subjects who remain under observation. For example, if patients in poorer health drop out of follow-up more often, censoring may be related to the likelihood of the event. In that case the test result can be biased.

Independence between observations should be considered.

Treating several hospital admissions of the same patient, or repeated contracts of the same customer, as independent observations can violate the independence assumption. For such data you may need methods that account for clustering or recurrent events.

Event times should be measured properly.

The log-rank test compares the number at risk and the number of events at each event time. So event times and censoring times need to be recorded accurately.

How should you think about the proportional hazards assumption?

The log-rank test is known to be an efficient test when the hazard ratio is constant. But the proportional hazards assumption does not have to hold for the log-rank test to be calculated or used.

Even when hazards are not proportional, the test of the null hypothesis can still be carried out under suitable conditions. However, if the direction of event risk changes over time or the survival curves cross, power can drop. So consider the shape of the survival curves together with the purpose of the test.

Things to watch out for with the log-rank test

When interpreting a log-rank test result, look not only at statistical significance but also at the actual survival probabilities and the characteristics of the data.

A significant p-value does not tell you the size of the effect.

With a very large sample, even a small real difference can come out statistically significant. So also check the difference in survival probabilities, the median survival times, and so on.

A non-significant result does not mean the two groups are the same.

With a small sample or too few events, power can be low even when a real difference exists. So you should not judge that two groups are the same from the p-value alone.

Be careful when survival curves cross.

The log-rank test can lose power when early and late differences cancel each other out. In such cases you can consider an RMST comparison or other weighted tests.

Handle censoring properly.

Subjects whose follow-up ended without the event should be treated as censored. Wrongly recording censoring as an event can distort the test result.

A difference between groups does not imply a causal effect.

The fact that annual-contract customers had a higher survival probability does not mean the contract type itself lowered the risk of cancellation. Other factors, such as why customers use the service, price, or satisfaction with the service, may also be involved.

You may need to account for competing events.

When another kind of event makes the event of interest impossible, an ordinary log-rank test alone may not be enough to answer the research question. In such cases you should consider competing risks analysis, the cumulative incidence function, and so on.

Where possible, also present the Kaplan–Meier survival curves and the survival probabilities at key time points to describe the direction and size of the difference between groups. Using the customer retention example from earlier, a write-up might look like this.

In real studies it is good to also present the number of events, the number censored, the number at risk, and so on. And if the survival curves may cross or the hazards may not be proportional, that should be reflected in the interpretation of the results.

If you compared three or more groups, it is best to report the overall log-rank test result and any needed post hoc test results separately.

Medical and health research

Compare the survival times, or the time to disease recurrence, of patients who received different treatments.

Marketing and customer analytics

Test whether the time until service cancellation differs by contract type or customer group.

HR and organizational research

Analyze whether the time until employees leave, or until they are promoted to a certain level, differs between groups of employees.

Education research

Compare how long students in different programs take to drop out or to complete the program.

Manufacturing and reliability analysis

Test whether the time until machine failure differs between parts or manufacturing processes.

Labor economics and social science research

Analyze whether the time until reemployment differs between population groups or between groups that did and did not take part in a policy program. As these examples show, the log-rank test is useful in studies where the time until an event and the differences between groups matter more than a simple event rate.

At each event time, it compares the number of events actually observed with the number expected under the null hypothesis, and combines these to calculate a chi-square test statistic and a p-value.

Where Kaplan–Meier estimation shows each group’s survival probabilities and survival curve, the log-rank test helps you judge whether there is a statistically significant difference in survival experience between groups.

But the log-rank test does not directly estimate the size or cause of the difference between groups. So you should also check the survival curves, the survival probabilities at key time points, the confidence intervals, and so on. And be careful when the survival curves cross or the direction of event risk changes over time, because power can drop.