this post was submitted on 23 Oct 2023
57 points (100.0% liked)

Science

13035 readers
27 users here now

Studies, research findings, and interesting tidbits from the ever-expanding scientific world.

Subcommunities on Beehaw:


Be sure to also check out these other Fediverse science communities:


This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[โ€“] Ranvier@sopuli.xyz 9 points 1 year ago* (last edited 1 year ago) (1 children)

Oh I agree. I'm coming at this from the medicine side though and there's some specific concerns and criticisms for Cochrane reviews and the meta analysis method in general, though on the whole I generally agree with what they're going for. A really simplistic example but I think illustrates the point well would be this:

https://www.bmj.com/content/327/7429/1459

A meta analysis of the effectiveness of parachutes concludes there is no evidence to suggest their effectiveness, due to lack of high quality double blind randomized controlled trials.

I'm a huge proponent of evidence based medicine, but sometimes people are quick to point to a Cochrane review or other systematic review as biblical scientific dogma that cannot be defied, and don't consider the nuances of the situation. Like what went into the meta analysis and why? What data might be missing from the meta review and why? Is only including double blind randomized controlled trials the best research method to answer the specific question being addressed? Was everything included together actually comparable? Was the specific question the review asked the right question to ask? How broad can this finding actually be applied?

Part of the problem in this case though was the Cochrane reviews findings being somewhat misrepresented in the popular media by those not familiar with the format or the nuances. No doctor is gonna read that review and rip off their masks when treating covid, flu, or other airborne or droplet diseases. Masks were used in the hospital long before covid and will continue to do so, for good reason.

what went into the meta analysis and why? What data might be missing from the meta review and why? Is only including double blind randomized controlled trials the best research method to answer the specific question being addressed? Was everything included together actually comparable?

The critiques / rebuttals to the masking review typcially ask these questions. This review almost falls apart after considering the answers.

If there's not enough information to form a solid conclusion, maybe they have no business analyzing it until there is. At the very least, if they're going to include partially unrelated studies, then reflect that in the title / opening statement. Don't say the analysis is of apples, then analyze fruit in general.