Skip to content

Reproducible research: A hunt for the truth

research-1029340_1920Last week, the journal Nature published the results of a survey asking scientists if they thought the published scientific literature is mostly correct.

The exact question they asked nearly 1,600 scientists in fields ranging from physics to biomedicine was, “How much published work in your field is reproducible?” Many scientists who answered the survey tended to be quite confident in their field's literature even though numerous studies have shown reproducibility as low as 11 percent in some fields. Three-quarters of the researchers thought that at least half of the papers published in their field would be reproducible.

But it’s not just Pollyannaish optimism that is the problem, say three researchers from the Meta-Research Innovation Center at Stanford (METRICS). It turns out that “reproducibility,” “replicability,” and several other terms are not used consistently in scientific communication. To fix the flaws of science, everyone needs to use such terms more thoughtfully and with precision, the researchers write in a paper titled “What does research reproducibility mean?” published today in Science Translational Medicine.

The three authors of the paper — Steven Goodman, MD, PhD; Daniele Fanelli, PhD; and John Ioannidis, MD, DSc — make the case that even if we define and use terms such as “reproducibility,” “replicability,” “reliability,” “robustness,” and “generalizability” consistently and correctly, what researchers are really after is the truth.

Goodman and his colleagues write that, “treating reproducibility as an end in itself — rather than as an imperfect surrogate for scientific truth — is partly responsible for the current terminological and operational morass, as well as how we can benefit by refocusing on cumulative evidence and truth.”

They include an amusing table of terms for misleading practices in science, including torturing, data snooping, and P-hacking.

“We need,” they write, “to move toward a better understanding of the relationship between reproducibility, cumulative evidence, and the truth of scientific claims.”

Previously: On communicating science and uncertainty: A podcast with John IoannidisAt the heart of reproducibility lies the problem of transparency and Re-analyses of clinical trial results rare, but necessary, say Stanford researchers
Photo by mwooten









Popular posts

Sex biology redefined: Genes don’t indicate binary sexes

The scenario many of us learned in school is that two X chromosomes make someone female, and an X and a Y chromosome make someone male. These are simplistic ways of thinking about what is scientifically very complex.
Intermittent fasting: Fad or science-based diet?

Are the health-benefit claims from intermittent fasting backed up by scientific evidence? John Trepanowski, postdoctoral research fellow at the Stanford Prevention Research Center,weighs in.