Friedman Test
A nonparametric test that compares, by rank, the measurements the same subjects show under three or more conditions.
If the same users tried three app screens in turn and then rated their satisfaction, which screen got the best ratings? If three treatment conditions were applied to patients, or the same students' learning achievement was measured at three time points, how do we check whether there was a change?
In this way, the method that compares, by rank, three or more measurements obtained repeatedly from the same subjects is the Friedman test. It tests for differences between conditions while accounting for the fact that the observations are paired.
Unlike the Kruskal–Wallis test, which compares three independent groups, the Friedman test ranks the conditions within each person. Whereas repeated measures ANOVA deals with differences in means, the Friedman test focuses on each individual's pattern of ranks.
It is useful in situations where the same subjects are evaluated under several conditions, as in user experience evaluation, education, and health research. In this article, we use a worked example to look at rank calculation, test results, post hoc comparisons, how it differs from other tests, and its assumptions and cautions.
What is the Friedman test?
A rank-based method for comparing the same subjects under several conditions
The Friedman test treats one participant's several measurements as a bundle (block), assigns ranks to the conditions within it, and checks whether the rank sums differ overall. Because it does not rely heavily on population normality, it can be used for ordinal responses or repeated measures data for which distributional assumptions are hard to justify.
What research questions is it used for?
First check that it is a repeated measures design, not independent groups
| Research setting | Conditions | Measurement |
|---|---|---|
| Comparing app screens | Rate all of screens A, B, and C | Satisfaction, 1–7 points |
| Comparing teaching methods | Three learning activities applied to the same students | Concentration rating |
| Medical research | The same patients assessed at three time points | Symptom severity grade |
What matters is that the observations in each condition must be linked to the same subjects. If the participants differ, you should consider an independent-groups analysis rather than the Friedman test.
How does the test work?
Rank within each person, then sum the ranks by condition
Among each participant's A, B, and C scores, the lowest value gets rank 1, the middle one rank 2, and the highest rank 3. If values are tied, the average of those ranks is used. Then all participants' ranks are summed by condition.
| Participant | Screen A | Screen B | Screen C | Ranks by condition |
|---|---|---|---|---|
| 1 | 4 | 5 | 6 | A=1, B=2, C=3 |
| 2 | 5 | 6 | 7 | A=1, B=2, C=3 |
| 4 | 5 | 5 | 6 | A=1.5, B=1.5, C=3 |
If a particular screen consistently gets high ranks from many participants, that condition's mean rank becomes larger. The actual Friedman statistic uses the differences in rank sums across conditions and is corrected when there are ties.
Real example: comparing satisfaction with three app screens
The same 18 participants rate all of screens A, B, and C
Eighteen hypothetical participants used all three navigation screens of an app and rated their satisfaction from 1 to 7 points. All three conditions are presented to every participant; in a real study, it is good to randomize or counterbalance the order in which the screens are presented to reduce order effects.
| Conditions | Responses | Median | Mean rank |
|---|---|---|---|
| Screen A | 18 | 4 | 1.03 |
| Screen B | 18 | 5 | 2.00 |
| Screen C | 18 | 6 | 2.97 |
How do you read the test results?
Check the test statistic, degrees of freedom, and significance level together
Analyzing the hypothetical data from 18 people with the Friedman test showed a statistically significant difference in the rank distributions of the three screens.
| Item | Result | Interpretation |
|---|---|---|
| Test statistic χ² | 35.00 | Summarizes the differences in ranks across conditions |
| Degrees of freedom (df) | 2 | 3 conditions − 1 |
| p-value | < .001 | Evidence against the null hypothesis of identical rank distributions |
This test alone cannot tell you between which two screens the difference occurred. Specific differences between conditions are examined with appropriate post hoc comparisons.
Between which conditions are there differences?
Pairwise post hoc comparisons suited to paired data
If the Friedman test is significant, you can run pairwise comparisons such as the Wilcoxon signed-rank test. Since several pairs are tested repeatedly, a multiple comparison correction such as the Holm correction is needed.
| Comparison | Raw two-sided p-value | After Holm correction | Interpretation |
|---|---|---|---|
| A vs. B | < .001 | < .001 | Screen B scores generally higher |
| A vs. C | < .001 | < .001 | Screen C scores generally higher |
| B vs. C | < .001 | < .001 | Screen C scores generally higher |
Pairwise p-values can vary with ties and the implementation of the test. The table summarizes, to about the third decimal place, a learning example that uses SciPy's approximate Wilcoxon method.
How do you describe the size of the difference?
Gauge the degree of agreement with Kendall's W
Apart from statistical significance, you should also look at the magnitude of the result. In repeated measures settings, Kendall's coefficient of concordance W is often reported. W generally lies between 0 and 1, and a larger value means the ranking of conditions is more consistent across participants.
The W for these data is about 0.97, which is very high. However, the practical meaning of an effect size should be judged in light of the scale and the study context, and it is best not to interpret it by arbitrary absolute cutoffs alone.
How does it differ from repeated measures ANOVA?
A difference in perspective: comparing means vs. ranks by condition
| Aspect | Repeated measures ANOVA | Friedman Test |
|---|---|---|
| Main values compared | Means by condition | Ranks by condition within blocks |
| Data | Continuous outcome variable | Ordinal or continuous outcome variable |
| Main assumptions | Normality of residuals, sphericity, etc. | Repeated measures pairing, independent blocks, comparable scale |
| Result | F, p, effect size | χ², p, Kendall W |
Rather than mechanically choosing between the two analyses based only on the results of a normality test, you should choose by considering the measurement scale, the study design, outliers, and the purpose of the analysis together.
What should you check before the analysis?
Pairing, missing data, ties, and order effects
Was each condition measured on the same participants?
You need to link the three conditions accurately by participant identifier so that data from different participants are not mixed together.
Are there missing measurements?
The basic Friedman test requires complete blocks across conditions. Automatically excluding participants with missing values can create sample selection bias, so you should examine the missing data pattern and alternative analysis methods.
What about ties and order effects?
Conditions with the same score get the average rank, and the test statistic should take the tie correction into account. When one person rates several screens, effects of learning, fatigue, and presentation order should also be controlled at the design stage.
How do you report the results in words?
Connect the sample size, test result, direction, and effect size
When the same 18 participants rated three app screens, there was a significant difference in satisfaction ranks by screen condition (Friedman χ²(2) = 35.00, p < .001, Kendall’s W = .97). Mean ranks were in the order of screen A (1.03), B (2.00), and C (2.97), and corrected post hoc comparisons also showed differences for every pair of screens (all Holm-corrected p < .001).
All figures presented here are values calculated from hypothetical data for learning. In an actual paper, the test options and post hoc comparison method used should be stated to match the analysis results.
How does it differ from other nonparametric tests?
Distinguish by independence and the number of conditions
| Method | Comparison design | Number of conditions or groups |
|---|---|---|
| Mann–Whitney U | Independent of each other | 2 groups |
| Wilcoxon signed-rank | Same subjects, two paired measurements | 2 conditions |
| Kruskal–Wallis | Independent of each other | 3 or more groups |
| Friedman | Repeated measures on the same subjects | 3 or more conditions |
If the same people rated the three screens, the Friedman test is appropriate, but if each screen was rated by different people, you should consider the Kruskal–Wallis test.
What should you be careful about?
Three things that are easy to forget when interpreting differences
A significant test does not mean that the differences in means of the three conditions were tested directly. It is important to also show the distribution of the raw data and the pattern of change by condition.
If the measurement order was always A → B → C, it is hard to separate the effect of the screens themselves from order effects. Also, a small p-value does not automatically confirm a causal effect of changing the screen.
If the data have very many ties or the measurements are extremely restricted, the rank information is reduced. In such cases, review the test method and the way of reporting more carefully.
Where can the Friedman test be used?
Many research settings in which the same subjects are evaluated repeatedly
It can be applied to the same students' ratings of three teaching methods they experienced, patients' pain grades under several treatment conditions, comparing users' satisfaction with three interfaces, surveys of the same raters' preferences for product prototypes, and so on.
The key is that the several conditions are obtained from the same subjects. Interpreting the direction and size of the results together while preserving the pairing of the study subjects lets you make the most of the Friedman test.
The Friedman test compares several conditions for the same subjects by rank.
Don't stop at the overall test result; look at the rank patterns and the post hoc comparisons as well.
The Friedman test is used to compare values for three or more conditions measured repeatedly on one person. Because it converts each individual's scores into ranks, it reduces differences in score levels between participants and lets you focus on the pattern of ranks by condition. A significant test is evidence that the conditions differ, but to find out between which conditions the difference occurred, separate post hoc comparisons are needed.
When reporting actual results, it is good to present the median and mean rank for each condition, the Friedman statistic and p-value, and the effect size together. You should also check whether the same person's responses were linked correctly, how missing data and ties were handled, and whether order effects arose during the repeated evaluations.
The key to the Friedman test is to convert three or more measurements obtained from the same subjects into ranks within each subject and examine the differences between conditions. When the result is significant, read the ranks by condition together with corrected post hoc comparisons, and interpret them with the repeated measures design and the characteristics of the data in mind.