Metaanalysis

Our last post highlighted a recent metaanalysis of active instruction in STEM courses, which leads the question—_what is metaanalysis?_ For a semi-technical review of the topic see here. This post will be much more general.

In yesterday’s post, each of the data points in the figures we reprinted was the result of the aggregated results of other studies. (Image Credit: Freeman et al. PNAS.)

On the left there are the results of 31 different studies that our combined into a single data point in Freeman’s metaanalysis on the right. As you might imagine, much of the difficulty in doing such an analysis comes in finding a way to put all of the different studies on equal footing. However, with the correct statistics there are several advantages to looking across studies:

• Provide more accurate and precise estimates of effect size.
• Identify inconsistent results in specific studies.
• Explore additional sources of variation in results.
• Investigate publication bias.

Caveat—combining many bad studies doesn’t magically give you a good meta-result. Data/study quality is always important! Also, metaanalysis is not immune to bias—researchers must still be careful and rigorous.

Meta-analysis has become an important part of medical research (particularly in identifying conflicts of interest), but perhaps the most public face of metaanalysis has been Nate Silver. In parallel with Sam Wang, Mr. Silver applied this idea of metaanalysis to polling data (weighting each poll by its historical accuracy) and predicted the outcomes of both the 2008 and 2012 presidential elections with almost perfect accuracy. You can read about this methodology here or here. He also applied these same ideas to predicting the most probable March Madness bracket.

The perfect bracket is astronomically unlikely. Perhaps we will cover this in another post.

Etymology

Meta- comes from the Greek μετα meaning “after” or “beyond”.

References

Meta. Wikipedia.

Meta-analysis. Wikipedia.

Freeman et al. Active learning increases student performance in science, engineering, and mathematics. PNAS.