Over at his blog Filling the Pail, Greg Ashman likes challenging popular ideas. In a recent post, he takes issue with meta-analysis as a way of analyzing educational research.
In the first place, Ashman argues — in effect — “garbage in, garbage out.” Combining badly-designed studies with well-designed studies still gives some weight to the badly-designed ones.
Of course, Ashman has some thoughtful suggestions as well.
Why Does It Matter?
Why should we care about such an obscure and complicated statistical technique?
Meta-analysis matters because we pay so much attention to it.
For instance: just a month ago, a pair of meta-analyses about Mindset Theory set off another round of anxiety. Edu-twitter lit right up with thoughtful scholars wondering if we should stop focusing so much on the right kind of praise.
Or: I frequently rebut claims about working memory training by citing this well-known meta-analysis by Melby-Lervag and Hulme.
If we’re going to rely so much on this technique, we should be clear-minded about its strengths and its weaknesses.
There is a great recent podcast about meta-analysis in the education research reading room which highlights some of the serious shortcomings of using it to push particular forms of interventions. You can hear it at https://goo.gl/4yrMnF .