Imagine recreating a recipe — say, your uncle Darryl’s BBQ sauce. Without a detailed recipe — including not only the list of ingredients but how long he cooked the sauce and when he added which ingredients — you’re not likely to be able to exactly reproduce his delicious sauce.
The same is true of research.
One of the frustrating recent discoveries about biomedical research is that, all too often, initially exciting results can’t be reproduced. And how do we know if an effect is real if we can’t make it happen again?
Researchers say that billions of dollars are wasted on research that has never been reproduced.
Lately, research funders and others have become obsessed with reproducible research. One problem is that most researchers aren’t motivated, financially or otherwise, to copy someone else’s study exactly. Better for your reputation and funding to do something original. The result is still another irreproducible study.
A second problem at the heart of irreproducible studies is transparency. Without a good understanding of a scientific paper’s methods, exact protocol and full results (which researchers rarely include in their paper), it’s very challenging for other researchers to reproduce the work.
Yesterday, Stanford’s John Ioannidis, MD, PhD, and his team at Stanford published one of two inaugural papers for the launch of PLoS Biology’s new Meta-Research Section. The paper by Ioannidis and colleagues provides a baseline that shows just how far we have to go to create a fully transparent biomedical literature.
Ioannidis, a professor of health research and policy, and colleagues took a random sample of 441 journal articles from biomedical journals from between 2000 and 2014. From that sample, they found that no team of authors made all their data available, only one team provided a full protocol, and the majority did not disclose funding or conflicts of interest.
With such low levels of transparency, it wasn’t surprising that replication studies were likewise rare. We can hope funders can find ways to motivate researchers to offer better transparency and reproducibility.
Previously: On communicating science and uncertainty, A conversation with John Ioannidis, “the superhero poised to save” medical research
Photo by jeffreyw/Flickr
