t-Test

Is there a difference in the average test scores of male and female students? Did students who took part in a new education program score higher than before they joined? Does a product's average satisfaction rating differ from 3, the neutral midpoint?

The t-test is the standard method for checking whether a difference in means is statistically significant. It is one of the most widely used statistical hypothesis tests in research papers, surveys, and experimental studies.

It is used in particular to compare the means of two groups, to compare two measurements of the same people, or to compare a sample mean with a specific reference value. In this article we'll look at the basic idea of the t-test, its three main types, a real analysis example, how to interpret the results table, and the assumptions to check before the analysis.

For example, suppose 100 students are split into two groups to compare two study methods. Group A uses the existing study method, and group B uses a new one. The test results are:

• Group A mean: 75 points
• Group B mean: 82 points

The two group means differ by 7 points. But the fact that the means differ isn't enough to conclude that the new study method is more effective. Test scores vary from student to student, and the means can also differ depending on how the sample was put together.

The t-test takes this sampling variability into account to judge whether the observed difference in means is statistically significant.

The standard error measures how much an estimate of a mean or a difference in means could vary if you drew samples repeatedly. The general idea of the t statistic is:

The actual formula depends on the type of t-test. For example, even if the difference between the two group means is 7 points, it may not be statistically clear if students' scores are very spread out. Conversely, if scores vary little, the same 7-point difference is more likely to come out statistically significant.

The t-test considers this difference and the variability together. In the results, you typically check the following measures.

All three methods compare means, but the appropriate one depends on how the data were collected.

One-sample t-test

A one-sample t-test checks whether a single sample mean differs statistically from a specific reference value. For example, suppose you want to know whether the mean service satisfaction, measured on a 5-point scale, differs from 3. Here you're not comparing two different groups; you're comparing one sample mean with the reference value of 3.

It is mainly used in situations like these:

• Checking whether a product's mean weight differs from the labeled weight
• Checking whether students' mean test score differs from a benchmark score
• Testing whether mean satisfaction differs from a preset target level

Independent samples t-test

An independent samples t-test compares the means of two independent groups. For example, you can apply a different study method to each of two groups made up of different students and then compare their test scores. The students in group A and group B must be different people, and you need to consider the independence of the observations.

There are two common versions of the independent samples t-test.

• Student's t-test: assumes the two populations have equal variances
• Welch's t-test: adjusted so it can be used even when the two populations have unequal variances

Consider Welch's t-test especially when the group variances differ or the group sizes are unequal.

Paired samples t-test

A paired samples t-test analyzes the difference between two linked measurements. A typical case is comparing values before and after an intervention in the same people. For example, suppose you measured 30 students' test scores before and after they took part in an education program.

Here you're not comparing two different groups; you calculate each student's before–after difference and test whether the mean difference is 0. The paired samples t-test can be used not only when the same people are measured twice but also when the study design pairs up two observations to compare.

Group A used the existing study method and group B used the new one. Each group had 50 students, and both took the same test after the training. The research question is:

Descriptive statistics by group

Comparing the two group means, group B, which used the new study method, scored 7 points higher on average. But to know whether this difference is statistically significant, you need to look at the t-test results.

Independent samples t-test results

Rounded results from Welch's independent samples t-test.

How do you interpret the results?

Group B's mean test score was 82, 7 points higher than group A's 75. The t-test gave a p-value of 0.003, smaller than the usual significance level of 0.05. So we can reject the null hypothesis that the two population means are equal and conclude that there is a statistically significant difference in the groups' mean scores.

The 95% confidence interval for the difference in means runs from about 2.43 to 11.57 points and does not include 0. The effect size, Cohen's d, was about 0.61. This means the difference between the two group means is about 0.61 times the pooled standard deviation.

To conclude that the new study method caused the score difference, however, you also need to review the study design, including whether assignment was random, whether the groups differed beforehand, and the learning environment.

If you set the significance level at 0.05, you generally call a result statistically significant when p is below 0.05. But statistical significance and practical importance are different concepts. For example, suppose you compared mean satisfaction with two services among 100,000 customers.

Mean satisfaction was 4.20 for service A and 4.22 for service B. With a very large sample, even a small 0.02-point difference can come out statistically significant. But whether a 0.02-point difference matters for running the service has to be judged separately.

Conversely, with a very small sample, even a fairly large difference in means may not be statistically significant. So with a t-test you shouldn't look only at the p-value; check the difference in means, the confidence interval, the effect size, and so on as well.

Cohen's d is the effect size most often used with t-tests. For an independent samples t-test, Cohen's d is usually calculated by dividing the difference between the two group means by the pooled standard deviation. The usual guidelines for interpreting it are:

These are conventional guidelines proposed by Cohen, not absolute standards that apply equally to every study. In medical research, for example, even a small difference can matter a great deal in practice, while in other fields even a fairly large difference may have little practical importance.

Also, effect sizes can be calculated differently, for example using the standard deviation of the difference scores in a paired samples t-test, so check which measure was used.

The most obvious differences are the number of groups being compared and the design of the analysis.

For example, to compare the means of two groups you can use an independent samples t-test. To compare the means of three groups that used three different study methods, on the other hand, you might consider one-way ANOVA.

In the standard equal-variance t-test comparing just two independent groups, the square of the t statistic equals the F statistic from a one-way ANOVA with the same design. But you shouldn't tell t-tests and ANOVA apart by the number of groups alone. ANOVA is also used for more complex study designs, such as repeated measures or analyzing the effects of several independent variables together.

The main ones are:

Independence of observations

In an independent samples t-test, the different observations must be independent. For example, treating the same students' before and after test scores as two independent groups ignores the link between the observations. In that case, a paired samples t-test is the right choice.

Normality assumption

The t-test relies on assumptions related to the normal distribution. The one-sample t-test considers normality of the population distribution, and the paired samples t-test normality of the distribution of paired differences. In the independent samples t-test, the population distribution of each group is one of the key assumptions.

That said, if the sample is large enough and there are no extreme outliers, the t-test can work fairly reliably even with some departure from normality.

Equal variances

The standard Student's independent samples t-test assumes the two populations have equal variances. When the variances differ, you can consider Welch's t-test. Because Welch's t-test doesn't require equal variances, it is sometimes used in practice as the default method for comparing independent sample means.

Checking for outliers

Extremely large or small values can change the mean and standard deviation a lot. So it's a good idea to check for outliers before the analysis using box plots, distribution plots, and the like.

But you shouldn't delete observations arbitrarily just because they are outliers. First check whether they are data-entry errors, whether they are values that could really occur, and how much they affect the results.

You need to consider the sample size, the shape of the data distribution, the extent of outliers, and so on. But if the data are very skewed or have many extreme outliers so that a mean-based analysis isn't appropriate, you can consider a nonparametric test.

Nonparametric tests, however, may not test exactly the same thing as a t-test. For example, the Mann-Whitney U test is basically a test of the distributions or relative location of two groups, and it can be read as a test of a difference in medians only under certain conditions.

So rather than switching methods mechanically based on the p-value of a normality test, consider the research question and the characteristics of the data together.

Using the study-method comparison above, you could write it up as follows.

Reporting results this way lets readers see not only whether the result was statistically significant but also which group had the higher mean, how large the difference was, and how uncertain the estimate is. In a real report, it's also best to state clearly which type of t-test was used and how the effect size was calculated.

Education research

You can compare test scores of two groups taught with different methods, or analyze changes in academic achievement before and after an education program.

Marketing and management

In an A/B test, you can compare the average purchase amount or satisfaction of customer groups that saw different ads or service screens.

Social science research

You can compare different groups' average stress levels, job satisfaction, life satisfaction, and so on.

Health and medical research

You can compare measurements before and after an intervention program, or compare the mean health indicators of two independent groups.

Products and quality control

You can check whether a product's mean weight or measurement differs statistically from a set standard. As these examples show, t-tests can be used, depending on the research aim, to compare with a reference value, to compare two independent groups, or to compare paired measurements.

The main types are the one-sample, independent samples, and paired samples t-tests, and the right one depends on how the data were collected and on the study design. And a statistically significant result doesn't necessarily mean the difference is important.

It's important to look at the difference in means, the p-value, the confidence interval, and the effect size together and interpret the results in light of the research aim.