Published by
Stanford Medicine

Category

Science Policy

NIH, Research, Science Policy, Stanford News

Shake up research rewards to improve accuracy, says Stanford’s John Ioannidis

Shake up research rewards to improve accuracy, says Stanford's John Ioannidis

currencyLab animals such as mice and rats can be trained to press a particular lever or to exhibit a certain behavior to get a coveted food treat. Ironically the research scientists who carefully record the animals’ behavior really aren’t all that different. Like mice in a maze, researchers in this country are rewarded for specific achievements, such as authoring highly cited papers in big name journals or overseeing large labs pursuing multiple projects. These rewards come in the form of promotions, government grants and prestige among a researcher’s peers.

Unfortunately, the achievements do little to ensure that the resulting research findings are accurate. Stanford study-design expert John Ioannidis, MD, DSci, has repeatedly pointed out serious flaws in much published research (in 2005 he published what was to be one of the most highly-accessed and most highly-cited papers ever in the biomedical field “Why most published research findings are false”).”

Today, Ioannidis published another paper in PLoS Medicine titled “How to make more published research true.” He explores many topics that could be addressed to improve the reproducibility and accuracy of research. But the section that I found most interesting was one in which he argues for innovative, perhaps even disruptive changes to the scientific reward system. He writes:

 The current system does not reward replication—it often even penalizes people who want to rigorously replicate previous work, and it pushes investigators to claim that their work is highly novel and significant. Sharing (data, protocols, analysis codes, etc.) is not incentivized or requested, with some notable exceptions. With lack of supportive resources and with competition (‘‘competitors will steal my data, my ideas, and eventually my funding”) sharing becomes even disincentivized. Other aspects of scientific citizenship, such as high-quality peer review, are not valued.

Instead he proposes a system in which simply publishing a paper has no merit unless the study’s findings are subsequently replicated by other groups. If the results of the paper are successfully translated into clinical applications that benefit patients, additional “currency” units would be awarded. (In the example of the mice in the maze, the currency would be given in the form of yummy food pellets. For researchers, it would be the tangible and intangible benefits accrued by those considered to be successful researchers). In contrast, the publication of a paper that was subsequently refuted or retracted would result in a reduction of currency units for the authors. Peer review and contributions to the training and education of others would also be rewarded.

The concept is really intriguing, and some ideas would really turn the research enterprise in this country on its head. What if a researcher were penalized (fewer pellets for you!) for achieving an administrative position of power… UNLESS he or she also increased the flow of reliable, reproducible research? As described in the manuscript:

[In this case] obtaining grants, awards, or other powers are considered negatively unless one delivers more good-quality science in proportion. Resources and power are seen as opportunities, and researchers need to match their output to the opportunities that they have been offered—the more opportunities, the more the expected (replicated and, hopefully, even translated) output. Academic ranks have no value in this model and may even be eliminated: researchers simply have to maintain a non-negative balance of output versus opportunities. In this deliberately provocative scenario, investigators would be loath to obtain grants or become powerful (in the current sense), because this would be seen as a burden. The potential side effects might be to discourage ambitious grant applications and leadership.

Ioannidis, who co-directs with Steven Goodman, MD, MHS, PhD, the new  Meta-Research Innovation Center at Stanford, or METRICS, is quick to acknowledge that these types of changes would take time, and that the side effects of at least some of them would likely make them impractical or even harmful to the research process. But, he argues, this type of radical thinking might be just what’s needed to shake up the status quo and allow new, useful ideas to rise to the surface.

Previously: Scientists preferentially cite successful studies, new research shows, Re-analyses of clinical trial results rare, but necessary, say Stanford researchers  and John Ioannidis discusses the popularity of his paper examining the reliability of scientific research
Photo by Images Money

Big data, Bioengineering, NIH, Research, Science Policy, Stanford News

$23 million in NIH grants to Stanford for two new big-data-crunching biomedical centers

$23 million in NIH grants to Stanford for two new big-data-crunching biomedical centers

More than $23 million in grants from the National Institutes of Health – courtesy of the NIH’s Big Data to Knowledge (BD2K) initiative – have launched two Stanford-housed centers of excellence bent on enhancing scientists’ capacity to compare, contrast and combine study results in order to draw more accurate conclusions, develop superior medical therapies and understand human behaviors.

Huge volumes of biomedical data – some of it from carefully controlled laboratory studies, increasing amounts of it in the form of electronic health records, and a building torrent of data from wearable sensors – languish in isolated locations and, even when researchers can get their hands on them, are about as comparable as oranges and orangutans. These gigantic banks of data, all too often, go unused or at least underused.

But maybe not for long. “The proliferation of devices monitoring human activity, including mobile phones and an ever-growing array of wearable sensors, is generating unprecedented quantities of data describing human movement, behaviors and health,” says movement-disorders expert Scott Delp, PhD, director of the new National Center for Mobility Data Integration to Insight, also known as the Mobilize Center. “With the insights gained from subjecting these massive amounts of data to  state-of-the-art analytical techniques, we hope to enhance mobility across a broad segment of the population,” Delp told me.

Directing the second grant recipient, the Center for Expanded Data and Retrieval (or CEDAR), is Stanford’s Mark Musen, MD, PhD, a world-class biomedical-computation authority. As I wrote in an online story:

[CEDAR] will address the need to standardize descriptions of diverse biomedical laboratory studies and create metadata templates for detailing the content and context of those studies. Metadata consists of descriptions of how, when and by whom a particular set of data was collected; what the study was about; how the data are formatted; and what previous or subsequent studies along similar lines have been undertaken.

The ultimate goal is to concoct a way to translate the banter of oranges and orangutans, artichokes and aardvarks now residing in a global zoo (or is it a garden?) of diverse databases into one big happy family speaking the same universal language, for the benefit of all.

Previously: NIH associate director for data science on the importance of “data to the biomedicine enterprise”, Miniature wireless device aids pain studies and Stanford bioengineers aim to better understand, treat movement disorders

Health Costs, Health Policy, In the News, NIH, Public Health, Science Policy

Research investment needed now, say top scientists

Top scientists made the case for continued investment in basic science and engineering earlier this week by unveiling a new report, “Restoring the Foundation: The Vital Role of Research in Preserving the American Dream” by the American Academy of Arts and Sciences.

Here’s why this is important: Federal investment is needed to power innovation engines like Stanford’s School of Medicine, and if that money gets funneled to roads, the military, Medicare, or any of a variety of other uses, fewer jobs, and fewer discoveries, could result. From the report:

Unless basic research becomes a higher government priority than it has been in recent decades, the potential for fundamental scientific breakthroughs and future technological advances will be severely constrained.

Compounding this problem, few mechanisms currently exist at the federal level to enable policy-makers and the research community to set long-term priorities in science and engi­neering research, bring about necessary reforms of policies that impede progress, or facilitate stronger cooperation among the many funders and performers of research…

Stanford President John Hennessy, PhD; biochemist Peter S. Kim, PhD; and physicist (and former U.S. Secretary of Energy) Steven Chu, PhD, are among the scientific rock-stars who co-authored the report.

For an excellent piece on the political debate surrounding the report’s release, check out the coverage in Science here. NPR also recently aired a series that colorfully illustrates the effects of research cutbacks, including a piece on a patient suffering from ALS, and a profile of several underemployed scientists.

Becky Bach is a former park ranger who now spends her time writing or practicing yoga. She’s a science writing intern in the Office of Communications and Public Affairs. 

Previously: More attention, funding needed for headache care, “Bold and game-changing” federal report calls for $4.5 billion in brain-research funding, Federal investments in research and higher education key to U.S. maintaining innovation edge

Podcasts, Public Safety, Science, Science Policy, Stanford News

The risks of tinkering with dangerous pathogens

The risks of tinkering with dangerous pathogens

In an effort to understand new and rare infectious diseases, researchers often use recombinant DNA technology to create novel strains in the lab. In 2012, researchers did just that, creating strains of the H5N1 influenza virus that were transmissible between mammals, setting off a debate about the ethics of creating viruses that were potentially more dangerous than those that occurred naturally.

Earlier this year, in July, a group called the Cambridge Working Group convened to continue discussing these questions. David Relman, MD, a biosecurity expert at Stanford, is a member of the group and spoke to Paul Costello about the risks and benefits of lab-created pathogens. Highlights of their conversation are in a piece in the most recent issue of Inside Stanford Medicine, where Relman notes:

My greatest fear is that someone will create a highly contagious and highly pathogenic infectious agent that does not currently exist in nature, publish its genetic blueprint, allow it to escape the laboratory by accident, or else enable a malevolent person or persons to synthesize the agent with the intention of releasing it in a deliberate manner. Although these may be unlikely scenarios, they could have catastrophic consequences, which is why I and others feel that we need to sensitize everyone to these possibilities and decide how to manage these risks ahead of time. I want to be clear: I am not opposed to laboratory work on dangerous pathogens, especially if they are known to exist in nature. Rather, I am opposed to high-risk experiments and, in particular, those that seek to create novel, dangerous pathogens that cannot be justified by well-founded expectations of near-term, critical benefits for public health — benefits that clearly outweigh the risks, and benefits that cannot be achieved through other means.

But not all researchers advocate the same level of caution. A few weeks after the Cambridge Working Group formed, another group called Scientists for Science to advocate in favor of using recombinant versions of pathogens in order to understand them better. Relman says that the two groups are probably not as far apart as they appear. He says he fully supports studying disease-causing bacteria, but:

The place where we may disagree is on whether we are willing to acknowledge that there may be experiments — probably few and far between — that perhaps ought not to be undertaken because of an unusual degree of risk. Just because a scientist can think up an experiment doesn’t mean it should be performed.

Relman elaborates on these topics in the 1:2:1 podcast with Costello above.

Previously:  How-to manual for making bioweapons found on captured Islamic State computer, Microbial mushroom cloud: How real is the threat of bioterrorism? (Very) and Stanford bioterrorism expert comments on new review of anthrax case

Applied Biotechnology, Bioengineering, Science, Science Policy, Stanford News

Stanford microscope inventor invited to first White House Maker Faire

Stanford microscope inventor invited to first White House Maker Faire

Foldscope-adams-squareLast week assistant professor of bioengineering Manu Prakash, PhD, received a call he couldn’t refuse — an invitation to attend the first-ever White House Maker Faire, to show attendees how to build a 50-cent microscope out of laser-cut paper, plastic tape and a tiny glass bead.

At today’s event, Prakash will also demonstrate how he turned a toy music box into a $5 programmable microfluidic chemistry set that can be used for applications as diverse as testing water quality and science fair projects.

Maker Faires, started by Make magazine in 2006, are gatherings where do-it-yourself enthusiasts show off their homemade projects and teach others how to make things using new technologies such as 3D printers, laser cutters, and desktop machine tools.

President Obama is hosting the first-ever White House Maker Faire to celebrate our “Nation of Makers” and to help empower America’s students and entrepreneurs to invent the future.

Prakash, who grew up in the mega-cities of India without a refrigerator, is a leader in the frugal maker movement. At Stanford, he works with students from bioengineering, medicine, and Bio-X to reengineer expensive, complex health-related devices to make them better, faster and cheaper.

His team also focuses on developing affordable science tools to inspire global innovation. To that end, Prakash recently launched an educational initiative called the “10,000 Microscopes Project,” where build-your-own-microscope kits will be shipped to the first 10,000 people who pledge to share their microscope images and experiments in a free, online microscopy manual.

“I’m so happy that the White House is looking at ways to celebrate scientific curiosity and invention,” Prakash told me. “Many children around the world have never used a microscope, even in developed countries like the United States. A universal program providing a microscope for every child could foster deep interest in science at an early age.”

For more information on the White House Maker Faire and today’s National Day of Making, or to watch the event live, visit www.whitehouse.gov/makerfaire or follow #NationofMakers on Twitter.

Previously: The pied piper of cool science tools, Music box inspires a chemistry set for kids and scientists in developing countries, Free DIY microscope kits to citizen scientists with inspiring project ideas and Stanford bioengineer develops a 50-cent paper microscope
Photo, of Quinn Monahan trying out a paper microscope, by Amy Adams
Photo in featured entry box by Manu Prakash

In the News, Neuroscience, Research, Science Policy

“Bold and game-changing” federal report calls for $4.5 billion in brain-research funding

"Bold and game-changing" federal report calls for $4.5 billion in brain-research funding

Collins and ObamaSome news today about the federal BRAIN Initiative – a major research plan aimed at revolutionizing our understanding of the human brain. (Stanford neurologist William Newsome, PhD, is co-chair of the initiative.) Francis Collins, MD, PhD, director of the National Institutes of Health, today heard and accepted a working group’s recommendations for the initiative’s budget and long-term scientific vision.

As outlined in an NIH release:

The report drafted by the ACD BRAIN Working Group maps out a sustained commitment of $4.5 billion in new federal funding over 10 years beginning in fiscal year 2016 to achieve seven primary goals. NIH already announced an investment of $40 million in fiscal year 2014 and President Obama has made a request for $100 million for NIH’s component of the initiative in his fiscal year 2015 budget.

The NIH efforts on the BRAIN Initiative will seek to map the circuits of the brain, measure the fluctuating patterns of electrical and chemical activity flowing within those circuits, and understand how their interplay creates our unique cognitive and behavioral capabilities.

Collins called the recommendations “bold and game changing” and noted, “As the Human Genome Project did with precision medicine, the BRAIN Initiative promises to transform the way we prevent and treat devastating brain diseases and disorders while also spurring economic development.”

Previously: NIH announces focus of funding for BRAIN initiative, BRAIN Initiative and the Human Brain Project: Aiming to understand how the brain works, Brain’s gain: Stanford neuroscientist discusses two major new initiatives, Co-leader of Obama’s BRAIN Initiative to direct Stanford’s interdisciplinary neuroscience institute, Experts weigh in on the new BRAIN Initiative and A federal push to further brain research
Photo, of April 2013 announcement of BRAIN Initiative, from the National Institutes of Health

Clinical Trials, Health Policy, Research, Science Policy, Stanford News, Videos

New Stanford center aims to promote research excellence

New Stanford center aims to promote research excellence

Updated 4-24-14: The center founders discuss METRICS in this just-posted 1:2:1 podcast.

***

4-23-14: Stanford has a new center, called the Meta-Research Innovation Center at Stanford, or METRICS for short, that will focus on ways to transform research practices to improve the reproducibility, efficiency and quality of scientific investigations.
When Stanford professor John Ioannidis, MD, DSc, discusses ideas on how METRICS might improve research quality, he points to the wealth of statistics within any newspaper’s sports section.

“Science needs as many ways to measure performance as sports do,” says Ioannidis. “More important, we need to find efficient approaches for enhancing this performance. There are many ideas on how to improve the efficiency of setting a research agenda, prioritizing research questions, optimizing study design, maximizing accuracy of information, minimizing biases, enhancing reporting of research, and aligning incentives and rewards so that research efforts become more successful. Possibly we can do better on all of these fronts.”

The center’s other co-director is Steven Goodman, MD, MHS, PhD, professor of medicine and of health research and policy.

METRICS’s core group of interdisciplinary scholars will be working on various aspects of meta-research, from methodologies to processes to policy. The center will also provide educational funding for students and scholars; organize collaborative working groups that include academics, policymakers, research funders and the public; and help establish similar initiatives worldwide.

You can learn more about “meta-research” and METRICS’s mission in the short interview above and in this release. Ioannidis discusses the center’s short- and long-term goals in the video clip below.

Previously: The Lancet documents waste in research, proposes solutions, “US effect” leads to publication of biased research, says Stanford’s John Ioannidis and Shaky evidence moves animal studies to humans, according to Stanford-led study
Photo in featured-entry box by Norbert Von Der Groeben

Public Health, Research, Science Policy, Videos

Federal investments in research and higher education key to U.S. maintaining innovation edge

Federal investments in research and higher education key to U.S. maintaining innovation edge

Government investment in research and higher education have made the United States a global innovation leader and have led to the creation of the Internet, global positioning systems, magnetic resonance imaging, touch-screen technology, and life-saving vaccines (among other things). But some worry that recent cuts and stagnating funding pose a serious risk to America’s ability to maintain its innovation edge at a time when other nations are rapidly increasing their research investments.

In preparation for the release of President Obama’s fiscal year 2015 budget proposal and the start of the appropriations season on Capitol Hill, a group of 14 business, higher education and scientific organizations have produced a video explaining the direct link between basic research, economic growth, improved medical treatments, and national security. Take a moment to watch it and learn more about how renewed investments in research would significantly benefit the country.

Previously: Future of medical research is at risk, says Stanford medical school dean, The economic benefits of publicly funded medical research, Report: NIH investments created $68 billion in economic activity last year, Academic medical centers bring billions to the economy and New initiatives show how federal stimulus dollars advance scientific and medical research

Medical Education, Medical Schools, NIH, Science, Science Policy

Medical school leaders to Congress: Stop NIH budget cuts

Medical school leaders to Congress: Stop NIH budget cuts

In a letter (.pdf) to Congress yesterday, nearly 200 medical school deans and hospital CEOs expressed their “grave concern regarding the impact of the continued cuts, especially those imposed by sequestration” on NIH-supported research. The group of leaders, including Stanford’s Lloyd Minor, MD, went on to say:

Sequestration already has resulted in the loss of $1.5 billion from the NIH budget in FY 2013. This reduction comes at the end of a decade that has seen NIH lose more than 20 percent of its purchasing power after inflation. As a result, the percentage of promising research proposals that NIH is able to fund has fallen to less than 17 percent, an all-time low. Furthermore, NIH estimates it will lose a total $19 billion from its budget if sequestration is allowed to continue for the next eight years, delaying progress for patients awaiting the chance for a better

Enacted and proposed cuts in NIH funding threaten current and emerging basic research opportunities across the country, as well as the clinical studies that are essential to bring scientific discoveries from the bench to the bedside. Further, these cuts also will discourage young people from careers in medical research, risking the loss of the next generation of innovators and their ideas.

Previously: Senate proposes to increase NIH’s budget in 2014, NIH director on scaring young scientists with budget cuts: “If they go away, they won’t come back” and Sequestration hits the NIH – fewer new grants, smaller budgets
Via Association of American Medical Colleges

Behavioral Science, Clinical Trials, Research, Science Policy, Stanford News

“U.S. effect” leads to publication of biased research, says Stanford’s John Ioannidis

"U.S. effect" leads to publication of biased research, says Stanford's John Ioannidis

pressuregaugeThe life of a scientist can be filled with pressure – pressure to publish, pressure to obtain funding, pressure to support the people in his or her lab. It’s no surprise that some would-be researchers *cough, me, cough* choose instead to pursue other careers (science writing FTW!).

Now new research by Stanford study-design expert John Ioannidis, MD, DSc, and Daniele Fanelli, PhD, from the University of Edinburgh, suggests that such pressures may lead to more than just sleepless nights. Their results, published today (subscription required) in the Proceedings of the National Academy of Sciences, conclude that, at least in areas of “soft” science that measure behavioral changes, studies with researchers from the United States are more likely to report that the intervention they were testing had an extremely positive outcome than studies whose authors hail from elsewhere. As Ioannidis explained to me last week:

One possible explanation for the pattern that we are seeing is that scientists from the United States are under higher pressure to generate extreme results. This could be for various reasons: to obtain funding, to justify a promotion or to advance one’s career.

Ioannidis isn’t suggesting that scientists are falsifying their results intentionally. Rather, some fields of research are more difficult to quantify than others. “In the behavioral sciences, results are presented with more leeway and creativity than in other ‘hard’ sciences like genetics, when, for example, there’s less room for error when sequencing a gene,” he explained.

For the study, the researchers analyzed more than 1,000 primary outcomes of 82 meta-analyses published in genetics or psychiatry published between 2009 and 2012. And, as Ioannidis points out, things could have changed since that time:

Our study shows what has happened in the past. It’s possible that this phenomenon is becoming more global. Unfortunately, there’s no “investigative pressure meter” we can use to directly compare the policies governing scientific research in various countries.

Previously: Shaky evidence moves animal studies to humans, according to Stanford-led study, Neuroscience studies often underpowered, say researchers at Stanford, Bristol and NIH funding mechanism “totally broken,” says Stanford researcher
Photo by William Warby

Stanford Medicine Resources: