Bayesian Analysis

How likely is it that a new training program is more effective than the existing one? Can we calculate the probability that a new product's purchase conversion rate is higher than that of an existing product? When study results come in, can we also take into account the information we already had?

A statistical approach that can be used for questions like these is Bayesian analysis. Bayesian analysis updates existing information or beliefs each time new data are observed, and expresses uncertainty about the parameters or hypotheses of interest as probability.

It differs from traditional frequentist statistics in how it interprets probability and in the process of statistical inference. It is used especially to express uncertainty about study results directly as probability, or to bring information from earlier studies into a new analysis.

In this article, we look at Bayesian analysis from its basic concepts to Bayes' theorem, prior and posterior distributions, a worked example, the Bayesian t-test and Bayesian ANOVA, and how it differs from conventional statistical analysis.

For example, suppose you want to find out whether a new study program raises students' test scores. Before starting the study, you may already have information from earlier research that teaching methods similar to this program can be somewhat effective.

Then, once you collect actual experimental data, Bayesian analysis combines the existing information with the newly observed data to update the estimate of the program's effect and its uncertainty.

In this process, the existing information is called the prior distribution, and the result updated after observing the data is called the posterior distribution.

The defining feature of Bayesian analysis is that it expresses uncertainty about parameters as a probability distribution. For example, if you have set up an appropriate model and prior distribution, you can answer questions like these.

• What is the probability that the effect of the new study program is greater than 0?
• What is the probability that the difference in means between the two groups is 5 points or more?
• What is the probability that the customer conversion rate is higher than a certain benchmark?

Bayes' theorem describes how to update probabilities by combining existing information with new evidence. In Bayesian statistics, the following relationship is generally used.

Each concept means the following.

prior distribution

The prior distribution expresses, as a probability distribution, the information you have about the parameter of interest before observing the current data. For example, if earlier studies found that the effect of a certain training program was generally small, you can reflect this by setting a prior distribution that gives relatively high probability to small effects.

You do not necessarily need earlier study results. When prior information is lacking, you can use something like a weakly informative prior that allows a wide range of values. However, no prior distribution can be declared completely neutral, so it is important to check the grounds for the choice.

Likelihood

The likelihood shows how plausible the actually observed data are, assuming a particular parameter value. For example, if you want to estimate a product's purchase conversion rate, you calculate how plausibly the actual numbers of visitors and buyers could arise at that conversion rate.

The likelihood plays the key role of bringing the information that new data provide about the parameter.

posterior distribution

The posterior distribution is the result of combining the prior distribution and the likelihood. It shows which parameter values are relatively plausible after observing the data and how much uncertainty there is about the parameter. In Bayesian analysis, this posterior distribution is used to calculate parameter estimates, credible intervals, the probability of meeting a particular condition, and so on.

The purchase conversion rate is the proportion of product page visitors who actually make a purchase. Before launching the new page, the store expected the conversion rate to be around 10%. It then collected data on 100 visitors and found that 15 bought the product.

Let's analyze these data with a Bayesian method.

Setting the prior distribution

Since the purchase conversion rate is a value between 0 and 1, we can use a beta distribution. For illustration, we set the prior distribution to Beta(2, 18). The mean of this prior is 0.10, which matches the prior expectation that the conversion rate will be about 10%.

Note that this distribution was chosen as an example; in a real study, the prior distribution should be chosen based on existing data and the purpose of the study.

Incorporating the new data

We observed that 15 of 100 people made a purchase. If we assume each purchase decision is an independent Bernoulli trial, the number of buyers can be modeled as following a binomial distribution. Combining the beta prior with the binomial likelihood gives the posterior distribution as follows.

In other words, the purchase conversion rate estimated by the posterior mean is about 14.2%.

How do we interpret the results?

We originally expected a purchase conversion rate of about 10%, but the actual data showed a conversion rate of 15%. Bayesian analysis updated the estimate of the conversion rate by combining the prior distribution with the observed data. As a result, the posterior mean came out to about 14.2%.

It is slightly lower than the observed 15% because the prior information that the conversion rate would be around 10% was also reflected in the estimate. This result alone does not let us conclude that the new product page is more effective than the existing page. Separate information about the existing page's conversion rate and its uncertainty is needed.

Also, the posterior mean is just one summary value. In real Bayesian analysis, it is important to use the whole posterior distribution to examine the uncertainty of the estimate as well.

A credible interval uses the posterior distribution to express the probability that a parameter lies within a certain interval. For example, suppose the 95% credible interval for the average effect of a training program is [2.1, 7.8].

This means that, given the chosen model, the prior distribution, and the observed data, the posterior probability that the training program's average effect lies between 2.1 and 7.8 is 95%. This should be distinguished from the confidence interval used in frequentist statistics.

The two intervals may be expressed as similar-looking numbers, but their statistical meanings differ. In particular, calculating a frequentist 95% confidence interval and then saying there is a 95% probability that the true parameter lies in that interval is not a strict frequentist interpretation.

For example, suppose you analyze whether a new training program is more effective than the existing one. A frequentist t-test usually sets up the null hypothesis that the two population means are equal and assesses how extreme the observed difference is under the null hypothesis.

A Bayesian analysis, on the other hand, can set up an appropriate model and prior distribution and calculate, for example, the posterior probability that the new program's average effect is greater than 0. However, the two approaches cannot be distinguished simply by which one is more accurate.

The appropriate approach depends on the purpose of the study, the characteristics of the data, the reliability of the prior information, and how you want to interpret the results.

The Bayes factor compares the relative evidence the observed data provide for two different hypotheses or models. For example, suppose we set up the following hypotheses about the difference in means between two groups.

• H₀: The difference in means between the two groups is 0.
• H₁: The difference in means between the two groups is not 0.

The Bayes factor BF₁₀ is the ratio comparing how plausible the data are under H₁ with how plausible they are under H₀. It uses the marginal likelihood of each hypothesis and is affected by the prior distribution set for H₁.

A Bayes factor of 10 does not mean the probability that H₁ is true is 90.9%. To calculate the posterior probability of a hypothesis, you need not only the Bayes factor but also the prior probability of each hypothesis.

How is it different from a p-value?

The p-value is the probability, given that the null hypothesis is true, of getting a result as extreme as or more extreme than the observed test statistic. The Bayes factor, on the other hand, compares how well two hypotheses explain the data relative to each other.

In particular, depending on the situation, the Bayes factor can show not only evidence supporting the alternative hypothesis but also evidence that relatively supports the null hypothesis. However, the Bayes factor is also affected by the chosen model and prior distribution, so you should not draw conclusions from a single number alone.

Like the ordinary t-test, it comes in several types depending on the study design.

For example, suppose we compare the test scores of 50 students taught with the existing study method and 50 students taught with a new study method. An ordinary t-test uses the t-value and p-value to judge whether the difference in means is statistically significant.

A Bayesian t-test can estimate the posterior distribution of the mean difference or the standardized effect size, or express the relative evidence for two hypotheses as a Bayes factor.

Example Bayesian t-test results

Actual results depend on the data and the prior distribution.

This result can be interpreted as follows: given the chosen model, the posterior probability that the group taught with the new study method has a higher mean score is 99.5%. Also, BF₁₀ = 8.4 means the data are about 8.4 times more plausible under the specified alternative hypothesis than under the hypothesis of no mean difference.

However, the posterior probability that the mean difference is positive and the Bayes factor answer different questions, so they should be interpreted separately.

Whereas ordinary analysis of variance (ANOVA) tests differences in group means using the F statistic and p-value, Bayesian ANOVA can compare several models or estimate posterior distributions for factor effects. For example, suppose we analyze the effect of three study methods on test scores.

• Group A: traditional lecture-based learning
• Group B: online learning
• Group C: discussion-based learning

Bayesian ANOVA can compare a model with no group effect and a model that includes a group effect.

Example model comparison

If BF₁₀ is 12.5, the observed data are 12.5 times more plausible under the model that includes the group effect than under the model without it. However, this result alone does not tell you which group's mean is highest or which groups differ.

To identify specific differences between groups, you need to run post hoc comparisons or analyze the posterior distributions of the differences in group means.

What if there are several factors?

Bayesian ANOVA can also be used in studies with several factors, as in two-way ANOVA. For example, when analyzing how study method and class format affect test scores, you can consider the following effects.

• Main effect of study method
• Main effect of class format
• Interaction effect of study method × class format

You can then compare the relative evidence for several models or estimate the posterior distribution of each effect.

It can incorporate existing information into the analysis.

Information from previous studies or past data can be expressed as a prior distribution. For example, if earlier studies have repeatedly found a similar effect, that information can be reflected in the analysis of a new study. However, if the existing information is inaccurate or does not fit the current study, it can adversely affect the results.

It can express results as probabilities.

You can calculate the posterior probability that the parameter of interest meets a particular condition. For example, you can calculate the probability that the new program's average effect is greater than 0, or the probability that it exceeds the smallest effect that matters in practice. These results can be used not only for statistical inference but also for decision-making.

It can express uncertainty as a distribution.

In Bayesian analysis, you can look at the posterior distribution rather than just a single estimate. This lets you see which values are relatively plausible and how much uncertainty there is in the estimate.

It can be extended to complex models.

The Bayesian approach can be applied to many kinds of statistical models, such as hierarchical models, repeated measures, handling of missing data, and time series analysis. In particular, when information for each group is limited, you can build a model that uses a hierarchical structure to partially share information.

Results can depend on the prior distribution.

The prior distribution plays an important role in Bayesian analysis. Especially when there are few data, the prior can have a substantial influence on the posterior. So it is good to explain clearly why you chose the prior and, if needed, to run a sensitivity analysis with different priors.

Computation can be complex.

For simple models, the posterior distribution can be calculated directly by mathematics. But complex models may require computational methods such as Markov chain Monte Carlo (MCMC). These methods can take a long time to compute, and you need to check whether the estimates have converged stably enough.

You need to check whether the model is appropriate.

In Bayesian analysis, too, using the wrong statistical model can lead to wrong conclusions. For example, if the assumptions about the distribution of the data or the independence of observations are not appropriate, the posterior distribution may not be trustworthy either.

So it is important to check how well the model explains the actual data, for example through posterior predictive checks.

Do not automatically interpret posterior probabilities as objective facts.

The probabilities calculated in Bayesian analysis are premised on the chosen model, the prior distribution, and the observed data. So a high posterior probability that a certain effect exists does not establish that the effect definitely exists in reality. The data collection process, study design, and appropriateness of the model must be considered together.

Do not draw conclusions from the size of the Bayes factor alone.

A large Bayes factor does not mean the hypothesis is absolutely true. The result can vary with how the compared hypotheses were set up and with the prior distribution, so you need to understand accurately what the relative evidence means.

Education research

You can estimate the effect of an educational program or bring earlier study results into a new analysis of learning effects.

Marketing and business

In A/B tests, you can calculate the probability that one product or ad has the higher conversion rate, or update information about customer behavior.

Health and medical research

You can estimate uncertainty about treatment effects or risks, and bring information from earlier studies into a new clinical study.

Social science research

You can analyze differences between groups, relationships between variables, hierarchical data structures, and more with Bayesian models.

Finance and economics

You can analyze uncertainty in parameters related to risk estimation, forecasting economic indicators, and investment returns.

Quality control and manufacturing

You can estimate product defect rates or the performance of a production process, and update existing estimates as new inspection results come in. In this way, Bayesian analysis is useful when you want to use existing information together with newly collected data, or interpret results in terms of probability.

For example, if you analyzed the difference in means between two study methods with a Bayesian t-test, you could report it as follows.

In a real study, you should also report which prior distribution was used, how the posterior distribution was calculated, and whether the model diagnostics were satisfactory. In particular, if you used MCMC, it is good to present information that shows the reliability of the computation, such as convergence diagnostics and effective sample sizes.

It works through the prior distribution, likelihood, and posterior distribution, and depending on the purpose of the study it can use posterior probabilities, credible intervals, Bayes factors, and more. Typical examples are the Bayesian t-test and Bayesian ANOVA, which can be used to analyze differences in means between groups or compare the effects of several factors.

However, Bayesian analysis is not a method that automatically replaces conventional statistical analysis. It should be chosen with the purpose of the study, the characteristics of the data, the reliability of the prior information, and the appropriateness of the statistical model in mind.