Meta-Analysis
Not just averaging the results of papers, but weighing precision and heterogeneity together.
How much does planting vegetation on city building rooftops lower indoor temperatures? One experiment reported an average drop of 0.6°C, while another reported 1.8°C. Which result should you believe? Because each study differs in building structure, climate, and sample size, it is hard to judge the general effect from a single study.
Meta-analysis is a method that statistically combines the effect sizes of several studies addressing the same question to estimate the overall effect and the differences between studies. The key point is that it doesn’t simply average the papers’ results; it takes into account the precision of the estimates and the heterogeneity between studies.
In this article we use six studies of how green roofs affect indoor temperatures in summer to cover effect sizes, inverse-variance weights, fixed-effect and random-effects models, heterogeneity, forest plots, publication bias, and reporting the results.
What is meta-analysis, and when is it used?
Meta-analysis quantitatively synthesizes effect estimates from several independent studies.
For example, you can gather studies that compared indoor temperatures in green roof buildings and conventional roof buildings and estimate the average temperature difference. You can also examine whether the direction and size of the study estimates are similar, and under what conditions larger effects appear.
Meta-analysis is related to the systematic review, but they are not the same concept. A systematic review is the entire process of defining a research question and then systematically searching, selecting, assessing risk of bias, and synthesizing the evidence.
Meta-analysis is the quantitative method used within that process when study results can be combined statistically. If there are not enough comparable studies or the data are unsuitable, it may be more appropriate not to do a meta-analysis even when you conduct a systematic review.
Worked example: the effect of green roofs on indoor temperature
Suppose there are six independent field studies that compared average indoor temperatures at similar times of day in summer.
Each study estimated the temperature difference between green roof buildings and comparison buildings; a negative value means the green roof building was cooler.
| Study | Mean difference (MD, °C) | Standard error (SE) | 95% confidence interval of each study |
|---|---|---|---|
| A | -1.20 | 0.30 | [-1.79, -0.61] |
| B | -0.80 | 0.25 | [-1.29, -0.31] |
| C | -1.50 | 0.40 | [-2.28, -0.72] |
| D | -0.60 | 0.30 | [-1.19, -0.01] |
| E | -1.10 | 0.35 | [-1.79, -0.41] |
| F | -1.80 | 0.50 | [-2.78, -0.82] |
These figures were created to demonstrate meta-analysis calculations.
All six studies report the temperature difference in degrees Celsius, so the mean difference (MD) can be combined as is. In a real analysis, though, first check the measurement conditions at each building, correlation from repeated measurements, and whether samples overlap between studies.
Several estimates from one study must not be included more than once as if they were independent studies.
How do you choose an effect size?
A meta-analysis needs a common effect size suited to the type of outcome.
For continuous outcomes measured in the same units you can use the mean difference (MD); if studies measured the same concept on different scales, consider the standardized mean difference (SMD, e.g., Hedges' g). For binary outcomes, the risk ratio (RR), odds ratio (OR), risk difference (RD), and similar measures are used.
| Outcome | Typical effect size | Example interpretation |
|---|---|---|
| Continuous, same units | Mean difference (MD) | Indoor temperature -1.0°C |
| Continuous, different scales | Standardized mean difference (SMD) | Difference in standard deviation units |
| Whether an event occurred | Risk ratio (RR), odds ratio (OR) | Relative ratio of event risk or odds |
| Correlation | Fisher z transformation of the correlation coefficient | Synthesizing correlations across studies |
RR and OR are usually analyzed after a log transformation and then exponentiated back for the results table.
What matters is making the direction and measurement time point of each study’s effect size consistent.
If you combine mean differences of -1.0°C and +1.0°C that were built on opposite sign conventions, the result will be wrong.
Why not give every study’s result equal weight in the average?
Each study’s estimate has a different amount of uncertainty.
In general, a study with a smaller standard error estimated the effect more precisely, so it gets a larger weight. The inverse-variance weight is calculated as wi = 1 ÷ SEi².
For example, study B has a standard error of 0.25, so its weight is 1/0.25² = 16. Study F has a standard error of 0.50, so its weight is 1/0.50² = 4. So in the basic fixed-effect calculation, study B carries four times the weight of study F.
These weights are based on a study’s precision. They do not automatically make a poorly designed study or one with a high risk of bias trustworthy. Risk-of-bias assessment is needed separately from the weight calculation.
How is the fixed-effect model calculated?
The fixed-effect model in meta-analysis assumes that all included studies estimate one common true effect.
The pooled estimate is then the inverse-variance weighted mean of the study effect sizes.
θ̂FE = Σ wiθ̂i ÷ Σ wi
Applying this to the six studies gives a fixed-effect pooled mean difference of about -1.03°C. This is an estimate that, under the common-effect assumption, indoor temperatures in green roof buildings are on average about 1.03°C lower than in comparison buildings.
In real studies, however, the effect itself may differ because of differences in climate, roof structure, and vegetation. In that situation you need to examine whether assuming a single true effect is appropriate. Also, the fixed-effect meta-analysis discussed here is a different concept from the firm or individual fixed-effects models of panel regression.
When is the random-effects model used?
The random-effects model assumes that the true effect can differ from study to study.
The between-study variance of the true effects is written τ², and the weights change to wi* = 1 ÷ (SEi² + τ²).
In this example, estimating the between-study variance with the DerSimonian–Laird method gives τ² ≈ 0.044. The random-effects pooled mean difference using this value is about -1.06°C. The conventional normal-approximation 95% confidence interval is [-1.38, -0.75]°C.
| Pooling method | Mean difference (MD) | 95% confidence interval | What it estimates |
|---|---|---|---|
| Fixed effect | -1.03°C | Can be calculated separately | A single effect shared by all studies |
| Random effects | -1.06°C | [-1.38, -0.75]°C | The mean of the distribution of study-specific true effects |
The numbers and confidence intervals here are examples to show how the calculation works. Because there are only six studies, a real report should consider uncertainty assessments suited to a small number of studies, such as Hartung–Knapp-type inference, along with the τ² estimation method.
The fact that a random-effects model reflects differences between studies does not explain the causes of the heterogeneity.
How do you read heterogeneity (Q, I², τ²)?
Heterogeneity indicates whether, and how much, the differences in study effect sizes go beyond simple sampling error.
You mainly check Cochran’s Q statistic, I², and τ².
I² = max( 0, (Q − (k−1)) ÷ Q ) × 100%
In the example, the number of studies is k = 6, Q ≈ 7.01, and the degrees of freedom are 5, so I² ≈ 28.7%. This value estimates the share of the observed variation in effect sizes that is related to real differences between studies; it does not mean a 28.7% temperature difference in individual buildings.
| Measure | Example result | What does it mean? |
|---|---|---|
| Q | 7.01 | Weighted sum of variation away from the common effect |
| I² | 28.7% | Estimate of the relative amount of variation due to between-study heterogeneity |
| τ² | 0.044°C² | Estimate of the variance among study-specific true effects |
When there are few studies, the Q test has low power and I² is highly uncertain. Rather than applying cutoffs such as 25%, 50%, and 75% as absolute standards, look at the content of the studies and the uncertainty of the estimates together.
How do you interpret a forest plot?
A forest plot shows each study’s effect size and confidence interval, along with the pooled estimate, at a glance.
Usually the size of each study’s point reflects its weight, and the horizontal line shows its confidence interval. The pooled result is often shown as a diamond.
In this example, the horizontal axis is the mean temperature difference. Values below 0 mean the green roof building was cooler, and 0 is the reference line for no difference. Study D’s confidence interval is [-1.19, -0.01]°C, very close to 0, while study F shows a wide confidence interval of [-2.78, -0.82]°C. This is why you need to read each study’s estimate and its precision together.
Even if the pooled effect is negative and its confidence interval excludes 0, you cannot conclude that a cooling effect will always appear in every setting. In a random-effects meta-analysis, a prediction interval, which also reflects the variation in study-specific true effects, can be useful as well, and interpretation needs particular care when there are few studies.
How do you explore the causes of heterogeneity?
If study effects differ, you can examine differences in climate zone, soil depth on the roof, vegetation type, study duration, and so on.
The main methods are subgroup analysis and meta-regression. For example, you can compare whether the effect differs between green roofs with thick vegetation layers and those with thin ones. Meta-regression estimates the relationship between such study-level characteristics and effect size with a regression equation.
In the current example with only six studies, however, a meta-regression with several predictors could be very unstable. Moreover, study-level relationships should not be interpreted as causal effects on individual buildings or residents. Heterogeneity analyses should be planned in advance, and it is best to state clearly that their results are exploratory.
How do you check for publication bias?
If results that are statistically significant or in the expected direction are more likely to be published than others, the meta-analysis result can be distorted.
This is called publication bias. More broadly, you should consider the problem of studies being selectively reported depending on their results.
A common visualization is the funnel plot, which shows the relationship between effect size and precision. If there are enough comparable studies, you can also consider methods such as Egger’s test to check for small-study effects.
But with only six studies, as in the current example, funnel plot asymmetry or Egger’s test is hard to interpret reliably. In particular, asymmetry can arise not only from publication bias but also from heterogeneity or differences in study methods. A nonsignificant test result should not be used as evidence that there is no publication bias, either.
How do you check whether one study drives the conclusion?
Sensitivity analysis is the process of checking how much analysis choices or specific studies affect the pooled result.
For example, you can run a leave-one-out analysis, removing studies A through F one at a time and recalculating the pooled effect.
You can also check the stability of the results by comparing the overall result with one that excludes studies at high risk of bias, or by applying other τ² estimators and confidence interval methods. Also check how each study’s effect size was computed and whether a shared control group was counted more than once.
However, you must not report only the analyses that produce the conclusion you want. Present the sensitivity analysis plan and its results transparently.
What determines how trustworthy a meta-analysis is?
Combining several studies does not, by itself, remove the errors or biases of the original studies.
Which studies were included and the methodological quality of each study have a large influence on the overall conclusion.
So in real research, define the research question and selection criteria in advance, and record the literature search strategy, search dates, and criteria for removing duplicate studies. Assess the risk of bias of the included studies, and check whether outcome measurement units, follow-up periods, and populations are sufficiently comparable.
For reporting, you can use systematic review reporting guidelines such as PRISMA 2020, and the certainty of the evidence can be assessed separately with approaches such as GRADE, depending on the research question. A statistically precise pooled estimate does not by itself mean high-quality evidence.
How do you report meta-analysis results?
When reporting meta-analysis results, present the research question, the search and selection process, the number of included studies, effect sizes, the pooling model, heterogeneity, uncertainty, and sensitivity analyses.
In random-effects analyses in particular, state clearly the τ² estimation method and the confidence interval method.
Sample report
To evaluate the effect of green roofs on indoor temperatures in summer, the mean differences of six independent field studies were synthesized. All studies found lower temperatures in green roof buildings than in comparison buildings. In a random-effects meta-analysis with the between-study variance estimated by the DerSimonian–Laird method, the pooled mean difference was -1.06°C, with a conventional normal-approximation 95% confidence interval of [-1.38, -0.75]°C.
Heterogeneity was estimated as Q = 7.01 (df = 5), I² = 28.7%, and τ² = 0.044°C². The results show an average association between green roofs and lower indoor temperatures, but because there are few studies and the measurement settings of the individual studies differ, caution is needed in generalization and statistical inference.
An actual paper should also report the risk of bias of the included studies, the search and selection process, confidence intervals suited to the estimation method, sensitivity analyses, and so on. If nonrandomized studies were synthesized, the pooled result should not be interpreted directly as a causal effect.
In meta-analysis, the quality and comparability of the evidence matter more than the number of studies
Meta-analysis statistically synthesizes the effect sizes of several studies to understand the overall effect and the differences between studies.
Because studies differ in precision, inverse-variance weights are used, and if the true effects may differ between studies, a random-effects model can be considered.
But obtaining a pooled result does not by itself settle the conclusion to the research question. You also need to examine the risk of bias in the original studies, how outcomes were measured, heterogeneity between studies, the possibility of selective publication, and the uncertainty of the estimates.