Crosstab Analysis and the Chi-Square Test

Do men and women prefer different products? Do the shopping channels people mainly use differ by age group? What is the relationship between education level and employment status?

The standard methods for checking how two categorical variables are related, as in these questions, are crosstabulation and the chi-squared test.

Crosstab analysis organizes counts and proportions for each combination of two variables in a table. The chi-square test decides whether the differences that appear are statistically significant. The two methods play different roles, but in real surveys and statistical analyses they are often used together.

They are especially useful for analyzing data divided into categories, such as gender, age group, occupation, product preference or purchase experience. In this article, we look at crosstab analysis and the chi-square test from their basic concepts through a real example, how to interpret the results table, effect sizes and what to check before the analysis.

The results are usually shown in a form called a crosstab or contingency table. For example, suppose you surveyed 200 customers about their gender and whether they prefer online shopping.

Gender is divided into men and women, and online shopping preference into 'prefer' and 'do not prefer.' Organizing the collected data in a contingency table gives the following.

The table shows that 60 of 100 men preferred online shopping, while 40 of 100 women preferred it. In other words, the rate of preferring online shopping was 60% for men and 40% for women. Crosstab analysis shows that the response distributions of the two groups differ.

But an important question remains.

To answer this question, you can use the chi-square test.

Frequency analysis checks how often each value appears in a single variable. Crosstab analysis, by contrast, combines two or more variables to compare distributions by category. For example, suppose you surveyed 200 customers about their preferred shopping channel.

Frequency analysis

Frequency analysis shows that half of all customers prefer online shopping. But it cannot tell you which customers prefer online shopping.

Crosstab analysis

Crosstab analysis lets you see how shopping channel preferences differ by gender. In other words, if frequency analysis shows the distribution of one variable, crosstab analysis shows the distribution across combinations of several variables.

Both analyses can be used as descriptive statistical methods for summarizing and understanding data.

Even for the same contingency table, what you are describing changes depending on whether you use row or column percentages. Typically you can look at row percentages, column percentages and total percentages.

Row percentage

Row percentages are calculated based on each row's total. If gender is set as the row in the earlier example, they look like this.

This table shows the share preferring online shopping within each gender group. So it suits the question, 'How do men's and women's shopping channel preferences differ?'

Column percentage

Column percentages are calculated based on each column's total.

In this example each column total is also 100, so the numbers come out the same. But the meaning is different: among people who prefer online shopping, 60% are men and 40% are women.

Column percentages are generally used to check the composition of the people who chose a particular response.

Total percentage

Total percentages are calculated based on the whole sample. For example, if 60 of all 200 people are men who prefer online shopping, the total percentage is 30%. Which percentage is more appropriate depends on the analysis question.

To compare differences in preference by gender, it is usually easiest to look at the proportions within each gender group.

When used with crosstab analysis, it mainly means the chi-squared test of independence. The chi-squared test of independence checks whether two categorical variables are independent of each other, that is, whether they are statistically related.

For example, if gender and shopping channel preference are independent, you would expect shopping channel preference rates not to differ systematically by gender. Conversely, if the two variables are related, the actually observed frequencies may differ from the frequencies expected under independence.

The chi-squared test of independence evaluates these differences statistically.

Hypotheses of the chi-square test

The following hypotheses are usually set.

The chi-square test computes a test statistic based on the differences between observed and expected frequencies. The chi-square statistic is written as follows. χ² = Σ (O - E)² / E

Here O is the observed frequency and E is the expected frequency. The larger the differences between observed and expected frequencies, the larger the chi-square statistic generally becomes. Statistical significance, however, is judged with the p-value calculated from the chi-square statistic and the degrees of freedom.

The observed frequency is the count found in the actual data. The expected frequency is the count expected under the assumption that the two variables are independent.

Let's use the gender and online shopping preference example from earlier again.

Actual observed frequencies

There are 100 men and 100 women in total. Also, 100 people prefer online shopping and 100 do not. If the two variables are independent, you would expect the share preferring online shopping to be the same in each gender, based on the overall distribution.

Expected frequencies are calculated as follows.

Calculating the expected frequency of every cell the same way gives the following.

Expected frequencies under independence

The observed frequencies differed between 60 and 40, but in the expected frequencies every cell was 50. The chi-square test evaluates whether such differences can be explained by chance sampling variation.

The research question is as follows.

Crosstab results

Values in parentheses are row percentages calculated within each gender group.

Chi-square test results

No continuity correction was applied.

How do you interpret the results?

While 60% of male respondents preferred online shopping, 40% of female respondents did. The two groups differed by 20 percentage points. The chi-square test gave χ²(1) = 8.00, p = .0047.

Because the p-value is smaller than the usual significance level of 0.05, you can reject the null hypothesis that gender and online shopping preference are independent. So you can interpret that there is a statistically significant association between gender and online shopping preference.

If the significance level is set at the usual 0.05, decisions are made as follows.

For example, p = .003 means that, assuming the two variables are independent, the probability of a difference at least as large as the one observed is about 0.3%. But a small p-value does not mean the relationship between the two variables is strong. With a very large sample, even a small association can be statistically significant.

Conversely, with a small sample, even a relatively large difference in proportions may not be statistically significant. So in a chi-square test it is important to look not only at the p-value but also at the proportions in the actual contingency table and the effect size.

For this you can use effect size measures such as the phi coefficient and Cramér's V.

Phi coefficient

The phi coefficient is an association measure used mainly for 2×2 contingency tables, where each variable has two categories. In the earlier gender and online shopping preference example, the phi coefficient was 0.20. This indicates how strong the association between the two variables is.

However, when interpreting the size of the value, consider the research field and the purpose of the analysis.

Cramér's V

Cramér's V is an effect size measure that can be used not only for 2×2 tables but also for tables with more categories. For example, dividing age into 20s, 30s, 40s and 50s and preferred brand into A, B and C produces a 4×3 contingency table.

Here too, you can use Cramér's V to check the size of the association between the two variables.

A Cramér's V close to 0 indicates a weak association, and values closer to 1 indicate a stronger association. However, interpretation criteria can vary with the number of categories in the two variables, the sample size and the research context, so it is best to avoid applying a one-size-fits-all standard.

But in a table with many categories, it can be hard to see which specific cells had more or fewer responses than expected. For example, suppose you compared four age groups and three brands.

Even if the chi-square test is significant, it does not tell you directly which age group chose which brand relatively often. In such cases you can check the standardized residuals or adjusted standardized residuals.

What is residual analysis?

A residual is the difference between the actually observed frequency and the expected frequency.

• Positive residual: the observed frequency is larger than expected
• Negative residual: the observed frequency is smaller than expected
• Residual near 0: observed and expected frequencies are similar

Standardized or adjusted standardized residuals let you compare how large the difference is in each cell. However, if you examine many cells at once, consider the multiple comparisons problem, and do not interpret a large residual as evidence of a causal relationship.

The main ones are the goodness-of-fit test, the test of independence and the test of homogeneity.

Chi-square goodness-of-fit test

Checks whether the observed distribution of one categorical variable matches a distribution set in advance. For example, you can test whether four product colors are chosen in equal proportions.

Chi-square test of independence

Checks whether two categorical variables are independent. It is used to analyze the relationship between two variables, such as gender and brand preference, or age group and service use.

Chi-square test of homogeneity

Compares whether the categorical response distributions of different groups are the same. For example, you can check whether the brand preference distributions of customers sampled separately in New York, Chicago and Houston are the same. The test of independence and the test of homogeneity use similar calculations on a contingency table, but they differ in the research question and the sampling design.

The chi-square test examined in this article alongside crosstab analysis is mainly the test of independence.

Check that the data are frequency data.

The chi-square test of independence generally uses observed frequencies for each combination of categories. It is not appropriate to enter means or percentages as if they were frequencies. Check that the analysis data represent frequencies of actual observations.

Check that observations are independent.

In the usual chi-square test of independence, observations must be independent of each other. For example, treating the same person's responses before and after using a service as two independent observations can cause problems. To compare the same subjects' before-and-after binary responses, McNemar's test or similar may be more appropriate.

Check that expected frequencies are not too small.

Depending on sample size and expected frequencies, the approximation using the chi-square distribution can become inaccurate. The traditional rule of checking whether expected frequencies are 5 or more is often used, but actual adequacy depends on the size of the table and the share of small expected frequencies.

So you should not judge solely by the simple rule that every cell's expected frequency must be 5 or more. When expected frequencies are small, you can consider Fisher's exact test or other exact or simulation-based methods.

Check that the categories are defined appropriately.

For example, dividing age groups too finely can leave very few observations in each cell. Conversely, forcibly merging categories with different meanings can make important differences disappear. So the criteria for dividing categories should be decided based on the research purpose and the characteristics of the data.

Check for missing values.

If survey responses are missing, the sample size used in the crosstab analysis can change. Before the analysis, decide whether to exclude missing values or treat them as a separate category. However, treating missing values as a response category is not always appropriate. Consider why the data are missing and the research purpose.

For example, to analyze the relationship between gender and brand preference, you could consider crosstab analysis and the chi-square test. To check the linear relationship between study time and test scores, on the other hand, you could consider Pearson correlation analysis.

Of course, there are also association measures for categorical variables that can carry a sign, such as the phi coefficient, and other association methods may be used depending on the measurement level of the data. So when choosing an analysis, consider both the measurement level of the variables and the research question.

Using the earlier gender and online shopping preference example, you could write it as follows.

Reporting results this way lets readers see not only the response distribution in each group but also whether there is a statistically significant association and how strong it is. In real research, it is also good to state, as needed, the sample size in each group, the basis for calculating percentages and how missing values were handled.

Marketing and management

You can analyze brand preference by age group, purchase experience by gender, service use by customer type and more.

Surveys and social science research

You can look at differences in the distribution of attitudes or choices by respondent characteristics such as gender, occupation and education level.

Education research

You can analyze preferred learning styles by grade, participation in and completion of educational programs, learning types by student characteristics and more.

Health and medical research

You can compare the association between lifestyle habits and particular health conditions, whether symptoms occur by group, categorical outcomes by treatment group and more.

HR and organizational research

You can analyze turnover intentions by department, training participation by job level, and the relationship between work arrangements and preferences for benefit programs.

Market research

You can check differences in product choice, ad awareness, purchase experience and brand preference across consumer groups. It is a basic analysis method especially for checking whether response rates in a survey differ by respondent characteristics.

The chi-square test of independence decides whether there is a statistically significant association between such categorical variables. Using the two methods together lets you see not only which responses appeared most in which group, but also how statistically distinct the observed differences in distribution are.

However, a significant chi-square test does not prove a causal relationship between the two variables. It is also important to look not only at the p-value but also at the actual frequencies and percentages and effect sizes such as the phi coefficient or Cramér's V.