Nonparametric Tests

What should you do if you want to check whether two groups differ in satisfaction but the data don't follow a normal distribution? What method should you use to compare 5-point scale responses collected in a survey, or to analyze data with a small sample and extreme values?

A statistical method you can consider in such situations is the nonparametric test. Nonparametric tests are statistical tests that assume no particular population distribution, or make fewer assumptions about the distribution than typical parametric tests.

They are used especially with ordinal data, heavily skewed data, and situations where you want to reduce the influence of outliers. In this article we look at the basic concept of nonparametric tests, how they differ from parametric tests, four representative nonparametric tests, a real analysis example, and points to watch when interpreting results.

The usual t-test and analysis of variance (ANOVA) are representative parametric tests that compare means. Nonparametric tests, on the other hand, include methods that compare the ranks of the data or test whether observations tend to fall in a particular direction. For example, suppose the class satisfaction of 100 students was measured from 1 to 5 points.

Here 1 point means very dissatisfied and 5 points means very satisfied. In such data the order of the values is meaningful, but you can't necessarily assume that the gap between 1 and 2 points is the same as the gap between 4 and 5 points.

For data collected on an ordinal scale like this, besides methods that compare means, you can consider nonparametric tests that use ranks. However, this does not mean nonparametric tests are more accurate or better for all data.

For example, if you want to compare the test scores of two groups, you can use an independent-samples t-test. On the other hand, if the test scores are heavily skewed, or you are comparing data such as satisfaction ranks instead of scores, you can consider the Mann-Whitney U test.

But you don't necessarily have to use a nonparametric test just because the normality assumption isn't met. With a sufficiently large sample, the t-test can work fairly reliably even when the data depart somewhat from normality.

Also, nonparametric tests don't always test a difference in medians. Depending on the method, they test hypotheses about population distributions, the relative position of ranks, the symmetry of differences and so on.

Each method can be distinguished by the number of groups being compared and whether the data are independent or repeated measures.

Mann-Whitney U test

The Mann-Whitney U test is a method for checking whether two independent groups differ in their distributions or relative position. For example, you can compare the satisfaction of students who took a class online with that of students who took it in person.

It converts both groups' responses together into ranks, then checks whether one group's ranks tend to be higher. It is often used as a nonparametric alternative to the independent-samples t-test, but the hypotheses the two methods test are not exactly the same.

Wilcoxon signed-rank test

The Wilcoxon signed-rank test is a method that analyzes differences between two paired measurements. It is typically used to compare values for the same people before and after taking part in a program. For example, suppose the learning confidence of 30 students was measured before and after a training.

Here you calculate each student's before-after difference, then test using ranks based on the direction and size of the differences. Unlike the paired-samples t-test, it does not require normality of the mean difference, but conditions such as a symmetric distribution of differences must be considered.

Kruskal-Wallis test

The Kruskal-Wallis test is used to compare three or more independent groups. For example, you can check whether the distributions of service satisfaction differ among customers in their 20s, 30s and 40s. It is widely used as a nonparametric alternative to one-way ANOVA.

However, even if the test result is significant, it does not directly tell you which groups differ, so additional post hoc comparisons may be needed.

Friedman test

The Friedman test is used when the same subjects were measured three or more times, or when comparing several related conditions. For example, suppose 30 customers were asked to use all three products and then rate their satisfaction. Because each customer rated all three products, the responses are not independent of each other.

The Friedman test takes this repeated measures structure into account and tests whether the ranks differ across conditions.

Satisfaction was surveyed among 50 customers who used Service A and 50 customers who used Service B. Satisfaction was measured on the following 5-point scale.

• 1 point: very dissatisfied
• 2 points: dissatisfied
• 3 points: neutral
• 4 points: satisfied
• 5 points: very satisfied

The research question is as follows.

Suppose the analysis used the Mann-Whitney U test, considering that the two groups are independent and the satisfaction responses are ordinal data.

Descriptive statistics by group

Service B's median was 4 points, higher than Service A's 3 points. However, the fact that the medians differ does not by itself let you conclude there is a statistically significant difference.

Mann-Whitney U test results

Actual results depend on the ranks of the individual responses, ties and the calculation method.

How do you interpret the results?

The Mann-Whitney U test gave a p-value of 0.008. This is smaller than the usual significance level of 0.05, which means you can reject the null hypothesis that the two groups' satisfaction distributions are the same.

Also, since Service B's median and quartiles were higher than Service A's, in this example you can describe Service B users' satisfaction responses as tending to be relatively higher.

So it is good to look at the response distributions as well as each group's median and quartiles.

For example, suppose data were collected from two groups as follows.

Pooling the values of both groups and ranking them from smallest gives the following.

Because there are two identical values of 4, both are given the average rank of 3.5. In this example, Group B contains more of the relatively high ranks. The Mann-Whitney U test uses the relative position of each group's ranks in this way to test whether the two groups can be regarded as coming from the same distribution.

Using ranks can sometimes reduce the influence of extremely large values on the result. For example, even if the value 10 changed to 1,000, as long as it stays the largest value, it can be reflected the same way in a rank-based calculation.

However, using ranks does not solve every outlier or data quality problem. You should also keep in mind that information about the gaps between actual values is reduced.

If they are two different groups

For example, if different customers rated Service A and Service B, the two groups are independent samples. In that case, if you are comparing the means of a continuous variable and the conditions for the t-test are appropriate, you can consider the independent-samples t-test.

On the other hand, if the responses are ordinal or a rank-based comparison suits the research purpose, you can use the Mann-Whitney U test.

If the same people were measured twice

For example, if the same customers rated their satisfaction before and after a service improvement, the samples are paired. If you are interested in the mean change and the appropriate assumptions are met, you can use the paired-samples t-test. If instead you want to analyze the ranks and directions of the differences, you can consider the Wilcoxon signed-rank test.

What matters is not just normality but considering together whether the samples are independent or paired, and what difference you want to test.

Kruskal-Wallis test: comparing several independent groups

For example, suppose you want to check whether service satisfaction differs by age group.

• Customers in their 20s
• Customers in their 30s
• Customers in their 40s

The customers in each age group are different people, so the groups are independent. If a rank-based analysis is appropriate, you can use the Kruskal-Wallis test.

Friedman test: comparing repeated measurements of the same subjects

This time, suppose 30 customers rated all three products.

• Satisfaction with Product A
• Satisfaction with Product B
• Satisfaction with Product C

Because each customer rated all three products, the measurements are related to each other. Using the Friedman test, you can check whether the rankings of the three products differ.

Both tests check for an overall difference. If a significant result appears, you can carry out additional post hoc comparisons to find specifically which groups or conditions differ.

For example, suppose the Kruskal-Wallis test gave a significant result. This means there is evidence against the null hypothesis that all groups have the same distribution. But it does not tell you which groups differ.

So you can carry out post hoc comparisons between groups. Representative methods are as follows.

For example, if you compared the satisfaction of customers in their 20s, 30s and 40s, you could carry out the following three comparisons.

• 20s vs. 30s
• 20s vs. 40s
• 30s vs. 40s

But running many tests can increase the risk of a Type I error. So it is good to also apply an appropriate multiple comparison correction such as the Holm or Bonferroni correction. When interpreting post hoc results, check the adjusted p-values together with each group's distribution, effect sizes and so on.

Even if a statistically significant difference appears, you need to check how large that difference is. The effect sizes used with nonparametric tests depend on the method.

For example, using the rank-biserial correlation with the Mann-Whitney U test lets you express how large the relative rank difference between the two groups is. With the Friedman test, Kendall's W lets you look at the degree of rank agreement across the repeated measurement conditions.

However, how effect sizes are calculated and the guidelines for interpreting them differ by measure. So when reporting results, clearly state which effect size you used.

Strengths

They can analyze ordinal data. They are useful for analyzing data where the order of values is meaningful, such as satisfaction levels or preference rankings. Some distributional assumptions weigh less heavily.

There are methods you can use even when a normal distribution is hard to assume. They can reduce the influence of outliers. Tests that use ranks can be relatively less sensitive to the size of extreme observations.

They can be used in many study designs. There are methods ranging from comparisons of independent groups to paired samples and repeated measures data.

Limitations

Information about the size of actual values can be reduced. Converting to ranks may not adequately reflect how large the gaps between observations are. They do not always have more power than parametric tests.

When the assumptions of a parametric test are adequately met, rank-based nonparametric tests can be relatively less efficient. Interpreting the results may not be simple. Some nonparametric tests do not directly test differences in means or medians, so you need to understand the results precisely.

They are not free of statistical assumptions. You need to check the conditions for each method, such as independence, the measurement level of the data and the symmetry of the distribution. So rather than thinking of nonparametric tests as unconditionally safer, it is important to understand the purpose and assumptions of each test.

Measurement level of the data

Check whether the variable you want to analyze is continuous, ordinal or categorical. For example, satisfaction rankings and simple nominal categories are different kinds of data. Rank-based nonparametric tests are not methods that can be applied as is to all categorical data.

Independence of observations

Methods for comparing independent groups and methods for analyzing repeated measures data must be kept distinct. Treating the same person's before and after measurements as independent observations can produce wrong results.

Shape of the distribution

If the two groups' distributions differ greatly in shape, it may be hard to interpret rank test results simply as a difference in medians. So it is good to also check boxplots or distribution plots for each group.

Sample size and ties

If the sample is very small or there are many ties, where the same value repeats, this can affect the calculation of the test statistic and p-value. Especially in data with a limited set of possible responses, such as 5-point satisfaction, many people may choose the same score. In such cases, check how the analysis method handles ties.

Missing values

If some observations are missing, the sample size actually used in the analysis can shrink. In repeated measures data, if values are missing at a particular time point, the number of subjects that can be analyzed may also decrease. So check the number of missing values and how they are handled before the analysis.

Using the service satisfaction example above, you could write it as follows.

Writing up the results this way shows not only whether there was a statistically significant difference but also how each group's responses are distributed. If possible, it is good to present an appropriate effect size and confidence interval as well.

Also, if you compared three or more groups or conditions, as with the Kruskal-Wallis or Friedman test, you should additionally report the necessary post hoc test results.

Survey and social science research

You can analyze group differences in ordinal responses such as satisfaction, attitudes and levels of awareness.

Education research

You can compare the satisfaction or learning attitudes of students who took part in different educational programs.

Health and medical research

You can compare ordinal measurements such as pain scores, symptom grades and patient condition ratings.

Marketing and business

You can check differences in preferences across customer groups, changes in satisfaction before and after a service improvement, and differences in ratings of several products.

Psychology research

You can analyze differences in ranked responses by experimental condition or in repeatedly measured rating scores. Nonparametric tests are worth considering especially when typical mean-based tests are not appropriate or when comparing ranks and relative positions is important for the research purpose.

Representative ones include the Mann-Whitney U test, the Wilcoxon signed-rank test, the Kruskal-Wallis test and the Friedman test. Each should be chosen according to the number of groups being compared, the independence of the data and whether there are repeated measures.

Also, nonparametric tests are not free of statistical assumptions, nor are they better than parametric tests in every situation. It is important to choose an appropriate analysis method by considering the shape of the distribution, the measurement level of the data and the research purpose together.