Power Analysis

When preparing a survey or an experimental study, one of the questions people worry about most is how many people's data to collect. Are 30 people enough? What about 100? Or do you need to survey 500 or more to get trustworthy results?

The sample size a study needs can't be decided simply as more or fewer. It depends on what statistical analysis you run, how small a difference or effect you want to detect, and how much certainty you want. The method used for this is power analysis.

Power analysis is a method that calculates the required sample size based on the size of the effect a study aims to detect and the statistical criteria, or evaluates the chance of detecting an effect with a given sample size. In this article we look at the concept of power analysis, what power and effect size mean, a real sample size calculation example, how it differs by analysis method, and points to watch.

Here, statistical power means the probability of correctly rejecting the null hypothesis when a real effect exists. For example, suppose you run an experiment to find out whether a new study method raises test scores more than the existing method.

Even if the new study method actually works, with too small a sample you may fail to detect the difference statistically. Conversely, with a sufficient sample, the chance of detecting the same effect goes up. Power analysis calculates this chance as a number to help design the study.

It is especially often used to decide the required sample size before a study begins.

That's because even when a difference or effect really exists, it can be hard to detect if the sample is small or the data vary a lot. In statistics, this situation is related to a Type II error.

For example, power of 80% means that if an effect of the assumed size really exists and the same study were repeated, you would expect about 80% of the studies to produce a statistically significant result. In the remaining roughly 20%, you might not get a statistically significant result even though a real effect exists.

These elements are connected to each other. For example, if the effect size and significance level are the same, power generally goes up as the sample size increases. Conversely, if the sample size is the same, the smaller the effect, the harder it is to detect.

Effect size

Effect size shows how large a difference between groups or a relationship between variables is. For example, knowing only that two groups' means differ by 5 points makes it hard to judge whether the effect is large. If the standard deviation of test scores is 5 points it may be a considerable difference, but if the standard deviation is 30 points it may be a relatively small one.

To compare such differences, standardized effect sizes like Cohen's d are sometimes used. The typical conventional interpretation guidelines are as follows.

These guidelines are not absolute standards that apply equally to every study. You should also consider the research field and the practical importance of the effect.

Significance level

The significance level is the criterion that sets how much Type I error you will allow in a statistical hypothesis test. 0.05 is commonly used, but a different value can be set depending on the research purpose and field.

For example, a significance level of 0.05 means the test procedure is designed to limit the probability of a Type I error to 5% when there is no effect. Setting a stricter significance level can increase the required sample size, other things being equal.

Statistical power

Power is the probability of detecting a real effect statistically when it exists. Many studies use 0.80 as the target power, and depending on the situation a power of 0.90 or higher is sometimes set. Securing high power reduces the risk of missing an effect, but it may require a larger sample.

Sample size

Sample size is the number of observations actually used in the analysis. A larger sample generally increases the precision of estimates and the power, but the cost and time of collecting the sample also increase. So it is important to calculate a sample size that fits the research purpose in advance.

In the study, students are split into two groups, each gets a different study method, and their final test scores are compared. The analysis method is the independent-samples t-test. Suppose the researcher sets the following conditions.

Running a power analysis with these conditions shows that you need about 64 people per group, 128 in total.

This is an approximate calculation that assumes the standard independent-samples t-test assumptions such as normality, independence and equal variances.

What does this result mean?

It means that if the effect size is 0.5, the significance level is 0.05 and an effect of the assumed size really exists, you can design the study so that power reaches about 80% with a sample of about 128 people. In an actual study, however, you may not be able to analyze every participant's data because of dropout or missing values.

For example, if a dropout rate of about 10% is expected, you need to increase recruitment so that 128 people remain for the final analysis. 128 divided by 0.90 is about 142.2, so by simple calculation you could consider recruiting at least about 143 people. However, the actual recruitment target should be adjusted according to dropout rates by group and the allocation method.

As in the previous example, assume a significance level of 0.05, power of 0.80, an independent-samples t-test, equal group sizes and a two-sided test. The approximate sample sizes by effect size are as follows.

The sample sizes are approximations based on standard power calculations and can differ depending on the calculation method and conditions used.

The table shows that the smaller the effect size, the more the required sample size increases. That's because detecting a small difference statistically requires more data. Conversely, if the effect size is large, you can detect the effect with a relatively small sample.

A priori power analysis

This approach determines the required sample size before the study begins. When the researcher sets the effect size, significance level and target power, it calculates the required sample size. It is the most practically used approach when writing a research proposal or designing surveys and experiments.

Sensitivity analysis

This is useful when the sample size you can obtain is already fixed. For example, if you can recruit at most 100 participants, you can check what size of effect can be detected with that sample size. This lets you review whether the study design is sufficient to detect the effect of interest.

Post hoc power calculation

This approach calculates power after the study is over, using the observed effect size, sample size and so on. However, post hoc power calculated by plugging in the observed effect size as is is closely tied to the p-value, so it may provide almost no additional information for explaining why a result was not significant.

This is why the same sample size standard cannot be applied to every study.

For example, an independent-samples t-test comparing the means of two groups and an analysis of variance comparing several groups calculate the required sample size differently. In repeated measures studies, the same subjects are measured several times, so the correlation between measurements must also be taken into account.

In regression analysis, the required sample size also depends on whether you want to test a particular regression coefficient or the effect of the model as a whole. So before running a power analysis, you should first decide on the research question and the statistical analysis method you will use.

The expected effect size must have a basis.

Setting a large effect size reduces the required sample size. But if the real effect is smaller than expected, you may not achieve the planned power. So set it reasonably, considering previous studies, pilot studies, the smallest practically meaningful effect size and so on.

High power does not guarantee research quality.

Even with a sufficient sample, the results can be less trustworthy if the sampling is biased, the measurement instruments have low reliability or the study design is inappropriate. Power is an important element, but it does not solve every problem in a study.

Account for missing values and dropout.

In surveys and long-term follow-up studies, some respondents may not take part or may drop out partway through. The sample size calculated in advance is generally based on the sample needed for the final analysis, so the expected dropout rate must be considered at the actual recruitment stage.

Check the independence of the sample.

When data are grouped in clusters, such as people in the same school or hospital, observations may not be independent of each other. For such clustered data, calculating the sample size based simply on the total number of people can underestimate the required sample size.

A statistically significant result and a practically important result are different.

With a very large sample, even a difference that is actually small can turn out statistically significant. So a study should consider not only the chance of detecting an effect but also how important that effect really is.

Social science research

Plan the number of respondents needed in a survey to examine relationships between variables or differences between groups.

Education research

Determine the appropriate number of participants when comparing the effect of a new educational program with an existing teaching approach.

Health and medical research

Plan the number of subjects, number of events and so on needed to test the effect of a treatment or intervention program.

Marketing and business

Estimate the sample size needed to detect a target effect in ad effectiveness experiments, customer satisfaction comparisons, A/B tests and more.

Psychology research

Plan the number of participants to analyze behavioral differences by experimental condition or relationships between psychological traits. Power analysis plays an especially important role when you need to explain the basis for your sample size in a research proposal or paper.

Without sufficient power, the chance of missing a real effect, even when it exists, goes up. Conversely, collecting far more data than the research purpose calls for can cost unnecessary time and money.

So it is important to decide on an appropriate sample size for a study by considering the analysis method, the expected effect size, the significance level and the target power.