John Ioannidis, MD, DSc, chief of the Stanford Prevention Research Center, has been involved in long-standing efforts to improve preclinical studies and the ways in which biomedical research is carried out and reported. In the process, he has authored several pieces examining weaknesses of the current system.
Now new research from Ioannidis and collaborators at Yale's School of Medicine and Oswaldo Cruz Hospital in Brazil shows that most medical interventions have only small or modest incremental effects, but that those effects are frequently overestimated by small studies.
In the study, which will be published tomorrow in the Journal of the American Medical Association, researchers examined data from the Cochrane Database of Systematic Reviews, a collection of review articles on health-care studies from around the world. Their analysis included trials that concluded a medical intervention had a very large treatment effect, defined as a positive or negative effect that was five times greater than the effect experienced by the control group. Overall, they scrutinized more than 3,000 reviews covering 85,000 meta-analyses on medical topics. Researchers' findings are explained by my colleague in a release:
The team discovered that in about 10 percent of the medical topics examined, a very large treatment effect was found in a first study, and another 6 percent found a very large treatment effect only in a later trial. But in more than 90 percent of all those cases, the very large effect disappeared when additional studies or meta-analyses were performed.
"Most of them did not survive to have a truly very large effect," said Ioannidis, who is also the C.F. Rehnborg Professor in Disease Prevention in the School of Medicine. And the very large effects usually didn't involve life-threatening aspects of the disease; rather, they were related to minor symptoms, or involved harms from the treatment being tested.
In fact, among all the interventions that the scientists studied involving randomized trials, only one intervention - a treatment known as extracorporeal oxygenation for respiratory failure in newborns - had a very large effect on mortality with high levels of statistical significance and no concerns about its validity. Overall, only 9 percent of the very large effects had high levels of statistical significance without any concerns about their validity.
Previously: Animal studies: necessary but often flawed, says Stanford’s Ioannidis, Outing bias in scientific research, A critical look at the difficulty of publishing “negative” results, Testing medical ‘truths’ and “Omics” studies need validation, says Stanford’s Ioannidis
Photo by Wellcome Library
