Last week, the journal Nature published the results of a survey asking scientists if they thought the published scientific literature is mostly correct.
The exact question they asked nearly 1,600 scientists in fields ranging from physics to biomedicine was, “How much published work in your field is reproducible?” Many scientists who answered the survey tended to be quite confident in their field's literature even though numerous studies have shown reproducibility as low as 11 percent in some fields. Three-quarters of the researchers thought that at least half of the papers published in their field would be reproducible.
But it’s not just Pollyannaish optimism that is the problem, say three researchers from the Meta-Research Innovation Center at Stanford (METRICS). It turns out that “reproducibility,” “replicability,” and several other terms are not used consistently in scientific communication. To fix the flaws of science, everyone needs to use such terms more thoughtfully and with precision, the researchers write in a paper titled “What does research reproducibility mean?” published today in Science Translational Medicine.
The three authors of the paper — Steven Goodman, MD, PhD; Daniele Fanelli, PhD; and John Ioannidis, MD, DSc — make the case that even if we define and use terms such as “reproducibility,” “replicability,” “reliability,” “robustness,” and “generalizability” consistently and correctly, what researchers are really after is the truth.
Goodman and his colleagues write that, “treating reproducibility as an end in itself — rather than as an imperfect surrogate for scientific truth — is partly responsible for the current terminological and operational morass, as well as how we can benefit by refocusing on cumulative evidence and truth.”
They include an amusing table of terms for misleading practices in science, including torturing, data snooping, and P-hacking.
“We need,” they write, “to move toward a better understanding of the relationship between reproducibility, cumulative evidence, and the truth of scientific claims.”
Previously: On communicating science and uncertainty: A podcast with John Ioannidis, At the heart of reproducibility lies the problem of transparency and Re-analyses of clinical trial results rare, but necessary, say Stanford researchers
Photo by mwooten