Interesting Links for 19-02-2013

[andrewducker]

• Did you know how hilarious the patch notes to the sims are
  (tags: errors programming thesims funny viaSkreidle )
• True Things About Alan Moore (a short comic, by Neil Gaiman)
  (tags: alanmoore comic NeilGaiman )
• 90% of all claims about the problems with medical studies are wrong
  (tags: medicine research science statistics )
• Canadian astronaut

Comments

[steer]

My knowledge of biology is near zero so I can't possibly judge the badness of your antioxidants example.

When you say "with no blinding whatsoever narrow that down to a dozen or so papers" I'm not sure what you're getting at by "no blinding". When I've read meta-reviews they take many papers from a keyword search as you suggest and get the results from them. To extract the results they must necessarily have read them and judge if they're actually relevant? I'm not sure what winnowing down process you are talking about to get from the 1,000 papers to the manageable size. I would guess if the original search throws up too many papers even to read abstracts you judge by title what to keep?

The step of contacting authors for unpublished results seems simply weird. What was the point of chucking out so many papers and then grabbing some extra results back in? I've never read a metareview paper that did this but I don't read too many. Is that a standard medicine thing or a standard Cochrane thing? My field is not medicine or biology and (being honest) I rarely read metareviews as my own field tends not to do them.

[steer]

Ah... thanks for the explanation. I guess I don't really care too much about the "blinding" part. I can see why medical types love it but most scientific disciplines manage without. You're right, it would be an "in an ideal world we would do this" thing but we've all got only finite resources available. Surely it's going to be pretty obvious if you drop very relevant papers because the authors dropped are going to notice. (And in my field they are likely to be the reviewers).

But this adding in extra stuff... that's just plain weird.

When I'm doing a review paper (not the same as meta review in the medical sense -- but a general tutorial, summary of a research area -- which admits bias for sure but by design) I would, if I were feeling particularly conscientious, contact an author to see if they agreed with my summary of what they said (especially when papers are not trivial to understand).

Thanks for your review of the meta-review -- it's interesting for sure.