Sunday, October 17, 2010

Lies, Damned Lies, and Medical Science

Here's an article from the Atlantic that everyone should read: Lies, Damned Lies, and Medical Science. It highlights the efforts of John Ioannidis to discover what's true and what's not true about modern medical research publications and clinical trials. I think this is going to become one of the hottest topics in science within a few years. The fallout will be horrendous when the public realizes that doctors are not as scientific as we thought.

Some interesting quotes from the article should prompt you to follow the link to the Atlantic website.
It didn’t turn out that way. In poring over medical journals, he was struck by how many findings of all types were refuted by later findings. Of course, medical-science “never minds” are hardly secret. And they sometimes make headlines, as when in recent years large studies or growing consensuses of researchers concluded that mammograms, colonoscopies, and PSA tests are far less useful cancer-detection tools than we had been told; or when widely prescribed antidepressants such as Prozac, Zoloft, and Paxil were revealed to be no more effective than a placebo for most cases of depression; or when we learned that staying out of the sun entirely can actually increase cancer risks; or when we were told that the advice to drink lots of water during intense exercise was potentially fatal; or when, last April, we were informed that taking fish oil, exercising, and doing puzzles doesn’t really help fend off Alzheimer’s disease, as long claimed. Peer-reviewed studies have come to opposite conclusions on whether using cell phones can cause brain cancer, whether sleeping more than eight hours a night is healthful or dangerous, whether taking aspirin every day is more likely to save your life or cut it short, and whether routine angioplasty works better than pills to unclog heart arteries.
Still, Ioannidis anticipated that the community might shrug off his findings: sure, a lot of dubious research makes it into journals, but we researchers and physicians know to ignore it and focus on the good stuff, so what’s the big deal? The other paper headed off that claim. He zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments such as the use of hormone-replacement therapy for menopausal women, vitamin E to reduce the risk of heart disease, coronary stents to ward off heart attacks, and daily low-dose aspirin to control blood pressure and prevent heart attacks and strokes. Ioannidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. That article was published in the Journal of the American Medical Association.

[Hat Tip, again to John Wilkins]


  1. I remember reading some of his papers a few years ago. I wouldn't be too concerned about drug trials, GWAS, and other types of studies where people try to find some associations, those are really expected to have a lot of garbage in them. What is more important to me is whether as one "hardens" the science and moves into cell and molecular biology, the same patterns hold. (My guess is yes but probably to a somewhat lesser extent).

    I can definitely see this being picked up and misused in all sorts of imaginable and unimaginable ways by the anti-science crowd...

  2. Management theory/fad is to Economics as Popular Medicine is to Biology.

    Of course some scientists would disagree, insisting economics itself is full of dubious claims. But for now the analogy will do.


  3. What's even worse is how much of this is due to outright corruption. The level of conflict of interest among faculty at the top US academic health centers is simply staggering. The general public is not really aware of this... yet.

  4. Still, Ioannidis anticipated that the community might shrug off his findings...

    Sure, another study might come along soon and contradict his result.

  5. I vaguely remember a couple of his papers some years ago. One of them was full of sweeping generalizations that I thought are almost certainly wrong.

    On the other hand, picking on modern "evidence-based" medicine is like shooting fish in the barrel. Everyone knows that it is wrong most of the time. No news here. Doctors are by far the worst scientists around and the fact that their subjects make good studies very difficult is only making things worse.

  6. I question the selection criteria for this analysis. Most widely cited does not equal most widely accepted. These articles may have been widely cited precisely because they were controversial.

  7. There is an inherent structural problem with medical research that means it can not be considered purely scientific. That problem is that it is incredibly difficult to publish medical papers with negative findings.
    Science makes progress, not by proving ideas correct but by figuring out which ideas are incorrect. Careers in medical research are made by publishing papers and that means essentially looking for things like small improvements over existing treatments (rather than showing that an existing treatment is useless compared to a placebo). Finding that a well established scientific model is, in fact, incorrect is very good science but a very bad career move. Criticizing a well established researcher, even if you have all the facts to back you up, is going to be very difficult to do in a peer reviewed journal. In a competitive field it is frequently your competitor who is the reviewer of the paper that criticizes his or her work. Anyone who works in medical research will be able to tell you many horror stories of such abuses of peer review (and the frequent complete lack of backbone of the editorial staff of many journals) in this regard.

  8. I have recently started a site, and the information you offer on this site has helped me a lot. Thank you for all of your time & work.