Published by
Stanford Medicine

NIH, Research, Science, Stanford News

NIH funding mechanism “totally broken,” says Stanford researcher

NIH funding mechanism "totally broken," says Stanford researcher

Stanford researcher John Ioannidis, MD, DSci, who is well-known for his critiques of much current scientific methodology, has now turned his sights on the National Institutes of Health. He and a colleague, Joshua Nicholson from Virginia Tech, have published an analysis (subscription required)  in today’s Nature questioning the way the organization funds research proposals.

Ionannidis and Nicholson argue that the peer-review process, in which groups called study sections review and rank research applications submitted by their colleagues, is inherently flawed and encourages “conformity, if not mediocrity.”

The result? Only 40 percent of scientists with highly cited papers are listed as the principal investigators on NIH grants. That is, those scientists whose peers value their insights and research most highly in their field are often not receiving federal support for that work.

The article is a pretty rousing condemnation of the status quo. Ioannidis, who is the chief of the Stanford Prevention Research Center, elaborated to me:

Our analysis shows with large-scale evidence that the system is totally broken. The majority of the US authors of the most influential papers in medicine and life sciences in the last decade do not have NIH funding; their funding rate may even be less than the rate of the average applicant. Conversely, study section members are almost always funded (a corollary of their selection process by NIH), but their citation impact is typically modest, nothing exceptional. High-impact innovators and funded study section members are almost completely mutually exclusive groups.

Ioannidis went on to note that the average age of a researcher receiving his or her first independent funding award from the NIH is 44 years (47 for MDs):

An out-of-the-box innovator who waits patiently and complies with orthodoxy until age 47 before becoming independent represents an oxymoron. A truly innovative idea cannot be judged by peers: if it is truly innovative, no peer has any clue about it; if peers already know about it, it is not innovative.

Although the NIH has tried to address concerns about a lack of support for new ideas with specific award categories such as the Pioneer and New Innovator Awards, Ioannidis called the outcome of such efforts “a drop in the bucket:”

We believe that NIH should fund some exceptional scientists based on the objectively measurable citation impact of their previous work. Peer review is the way it is applied now encourages conformity and mediocrity and favors people who know how to network and play the petty games of academia, not those who have brilliant ideas.

Previously: Research shows small studies may overestimate the effects of many medical interventions, Animal studies: necessary, but often flawed, says Stanford’s Ioannidis and Outing bias in scientific research
Photo by videocrab

10 Responses to “ NIH funding mechanism “totally broken,” says Stanford researcher ”

  1. Jeremy Berg Says:

    The claims of this work greatly exceed the analysis that appears to have been performed. The conclusion that “the system is totally broken” seems entirely unjustified. The analysis states that 40% of the scientists with highly cited papers are listed as principal investigators on NIH grants. Key questions appear to be unaddressed such as: What percentage of these investigators have applied for NIH funding? Do they have other sources of support (such as HHMI) that would power a substantial research program and would discourage them for needing NIH support?

    The notation that NIH should fund some exceptional scientists based on their previous citation impact requires considerable analysis. Citation rates vary substantially from field to field and some investigators have one highly cited paper without making additional outstanding contributions.

    I am not arguing that the NIH system is ideal or highly suited to taking important risks, but I do think the system (including the role of NIH staff, not just peer reviewers) is certainly not “totally broken”.

    In the spirit of disclosure, I served as Director of the National Institute of General Medical Sciences at NIH for nearly eight years and was deeply involved in establishing and running the NIH Director’s Pioneer and New Innovator programs. I am also an active scientist.

  2. John Ioannidis Says:

    Dear Jeremy, thank you for your interest in this analysis. I fully agree with you that more evidence and more analyses would be very useful. Nevertheless, based on what we see, the large majority of these non-funded stellar scientists are not funded by HHMI (which is doing its very best, but with a much small portfolio than NIH), and many of them just move out of academia or even out of science all together. If we encourage this situation, then certainly more of these scientists will never be able to produce another such influential paper.

    I know of no field where a paper with 1000 or more citations would not be considered extremely influential. I acknowledge that there can be subjectivity in interpreting the same numbers sometimes, but I think it is important to generate numbers rather than just opinions. Perhaps you feel that 40% is not bad enough yet, and we should wait to use the term “totally broken” only when we reach the point that 0% of the scientists producing the most influential papers receive NIH funding subsequent to their milestone achievement. Or, perhaps you think that 44 or 47 as an average age for the first independent investigator award is also not bad enough, and we should use the term “totally broken” only when scientists in this country first become independent when they are 67 years old.

    At any rate, I think we agree that science is the most noble human endeavor and we should do our best to support it with more funds and more efficiently. In this regard, you are my hero is being instrumental in promoting the Pioneer and New Innovator programs and I hope your voice continues to be heard in trying to support the best of the best and not the most networked of the most networked.

    Kind regards,
    John Ioannidis

  3. Jeremy Berg Says:

    John: With regard to the 40% figure, I feel it would be more useful to have full context for this figure. This could be interpreted by some to mean that these scientists apply to NIH and only 40% receive funding. This may be try (although I am inclined to doubt it). More likely a significant fraction of these individuals have moved in different career directions for a variety of reasons. Some of my best students have gone on to careers outside of academia where they are quite successful but publish relatively little and never apply for NIH support.

    I absolutely agree with you that providing quantitative analyses that can be debated is better than purely subjective discussion. Again, however, context is important. Consider, for example, the age of “new investigators” receiving NIH funding. Let me make several comments. First, these are “new investigators” (those who have not previously substantial NIH support) and include both new Assistant Professor but also senior investigators who have moved from overseas or who have been supported by NSF or other agencies). Looking at new Assistant Professors funded by NIGMS, the median age was between 36 and 38. I agree that this is too old (Full disclosure: I was an Assistant Professor at 28 and a Department Chair at 32) but it is much better than 44 to 47. Furthermore, this age appears to have been relatively stable for a decade (after a substantial rise). A major driver for this age is not NIH directly but biomedical research academic culture. Many starting scientists complete 5 or more years of postdoctoral training prior to beginning their independent careers and then wait 2 or more years before submitting their first NIH application. I personally do not believe that such long periods are necessary for training and may be detrimental to boldness in research.

    With all of that said, I agree that NIH review can be frustratingly conservative. One of the lessons that I learned from the Pioneer and New Innovator programs is that review groups that are both selected for their apparent risk tolerance and (perhaps more importantly in my opinion) being explicitly charged with looking for bold ideas can perform fairly well. However, review groups that must compare bold but risky proposals with more incremental but solid and important proposals struggle and usually take the more cautious position. This is likely exacerbated in tough budgetary times such as the present.

    Finally, I think we all need to be careful about how we communicate the analyses of NIH processes. The results (as are many results in science) are often nuanced and having headlines in the popular press about how broken the NIH review system is probably not helpful.

  4. John Ioannidis Says:

    Dear Jeremy: I think we are starting to converge and we share many common experiences, not just the fact that we were both Department chairs in our early 30s. But I feel embarrassed, if not ashamed, about my achievements, when I need to convince my trainees that at the same age when I became department chair they should think about starting a new training period of another 5, 10, or 20 years before they can become (almost, not really) independent. It is indeed possible that some of the most talented people don’t even apply for NIH funding at all. But then is this reassuring? This would only prove that the situation is worse: we have managed to create a system where academic research is no longer attractive to the very best, it has become so onerous to their eyes that they are not willing even to take a shot. Once these very best people have left academia, research, or both, what kind of monster successful academic professor are we breeding? This surviving species may be more suitable for working in Wall Street than in science.

    I am not sure where you see that the age of first award as principal investigator has been relatively stable in the last decade. The data suggest a continuously growing age over this time period (http://nexus.od.nih.gov/all/2012/02/03/our-commitment-to-supporting-the-next-generation/). At the risk of highlighting subgroup analysis, the situation is getting worse for MD applicants in particular. Blaming this trend to investigators who invade NIH from NSF and from overseas is a bit like blaming aliens from Mars. Funding and funding agencies outside of the US/NIH may be even worse, I think this is worth of empirical scientific study. Science is a global enterprise without disciplinary boundaries and I know of many superb investigators from outside the US who have been amazingly productive and innovative with miserable salaries and no grant support at all. I bow with great respect to their perseverance and not funding-dependent scientific ethos. At any rate, I am happy we eventually agree that the age of successful applicants is “too old” and that “NIH review can be frustratingly conservative”. I could not have found better words to describe the situation.

    How evidence on NIH processes or any scientific processes should be communicated to the public is a tantalizing question. My viewpoint is (and I may well be wrong) that science is about getting to the truth and thus it is natural to convey the truth in full also to the public. This requires full disclosure of our failures as scientists, rather than trying to show that everything we do is perfect. Science is currently under attack by a coalition of dogma: religious, denialist, political, intelligent design, etc. The common denominator of all these dogmas is that they all believe they are perfect, and that their processes are flawless. I prefer to take the side of science, because I personally make mistakes all the time and I am interested in identifying mistakes and correcting them. If something is broken, we need to first acknowledge it is broken, if we ever want to fix it. Interestingly, in today’s Science the PCAST report is discussed and I could not agree more with what is says: “This year, it notes, NIH will make 50 awards in the three director’s categories out of a total of 35,944 research grants. While this plethora of initiatives, each worthy in its own way, gives an illusion of significant progress, the report notes, in truth the sum of all these programs is tiny, almost invisible, in comparison to each agency’s dominant form of research support.” Kind regards, John

  5. Jeremy Berg Says:

    Hi John: I agree that academic careers are in danger of becoming sufficiently unattractive that we are likely losing talented people to other careers. My point is that I feel this is largely the doing of the academic community rather than NIH or NIH policies. My personal view is that postdoctoral training periods have become excessively long and that professors have a huge conflict of interest since having talented and experienced scientists driving their research programs is quite attractive if you fully consider the impact on the “trainees”. The growing unattractiveness of academic careers is a great concern to me.

    With regard to the age of NIH new PIs, if you look at the chart in your link, the curve for Ph.D.s has been constant near 42 since 2001. Since Ph.D. make up over 70% of the NIH PIs, I believe that this implies that this is approximately true for NIH New PIs overall. As you note, the situation is worse for MDs and MD, PhDs. This gets into changes in MD training which opens up another whole set of issues. With regard to “blaming” scientists who have been supported by other agencies, my point was that it is worth disaggregating NIH “new investigators” into early career new investigators (e.g. starting assistant professors) and other new investigators (full professors who are new to NIH). FIve years ago, “new investigators” approximately equal numbers of the two groups and they have different stories.

    With regard to the communications issues, I did not mean to advocate implying that everything is perfect., far from it. However, the general public does not understand the NIH peer review system at all and could conclude that it has no merit. The peer review system is actually quite robust compared to many alternatives. It is important to find fora where these issues can be discussed without losing important nuance.

    With regard to the last statistic, I agree that the number of explicitly innovative awards is too low. But this statistic is misleading. The 50 awards in the “director” category refers only to new competing awards while the 35,944 includes non-competing awards. There are approximately 60 active Pioneer Awards and approximately 250 active New Innovator Awards. Again, the number is too low, but quoting statistics that are systematically off by a factor of five does not improve the clarity of the discussion.

  6. Joshua Nicholson Says:

    Dear Dr Berg,

    I am glad to see a fruitful discussion on the topic of science education and funding. I think the best way forward is to openly discuss current practices and potential alternatives to funding and training as you and John are. As a current graduate student I think daily about the road ahead and if it is really worth it to pursue an independent scientific position. What will funding be like in 5 or 10 years? Will the success rate be even worse then? Can I start a family on a postdoc salary and job security? Will my criticisms of the NIH hurt my chances of being funded by them?

    I think you are correct that the academic community is in part to blame for the current unattractiveness of a scientific career, but I think this is very strongly linked to funding. PIs keep postdocs and students increasingly longer than they did in the past to further their own career and better increase their chance of funding. Post docs and students stay because they cannot compete on their own.

    To improve the situation I think that the NIH should implement multiple forms of review, since all will have shortcomings in one way or another. As it currently stands, the structure of study sections and peer review at the NIH strongly limits new or different ideas. The fact that study section members work is very close to the applicants they review and that study section members rely on the same pool of money as the applicants sets up a direct conflict of interest. Why would a researcher studying an idea or process for many years/decades fund an idea different from their own? This would hurt their own career and reputation. In my mind, this means that if I want to get funding I must pursue fashionable and safe ideas and frankly I think this kills the passion of students. It is as if we are selecting for sycophancy in students. Indeed, if I were to write a grant today I would try and cite members of the study section in hopes that that betters my chances of funding.

    Would it not be good to maintain the current system but have other forms of review? Perhaps some where study section members are made of scientists from different disciplines than the applicants. Perhaps some where reviewers are blinded. Perhaps some where a certain criteria (1000 cited researcher) are automatically given money. I am not sure if these would necessarily be an improvement, but I think they are worth testing and I hope that our comment may spark this into action.

    Best regards,
    Josh

  7. John Ioannidis Says:

    Dear Jeremy: I think we are converging, at least we are looking at the same numbers, and we are both not happy with them. <1% is nowhere close to be called a big proportion. I am not trying to blame the NIH. The NIH, reviewed scientists and reviewing scientists are the same interdependent community, we are all in the same boat and we should keep it afloat. I am not certain about the statement that “the peer review system is actually quite robust compared to many alternatives”. It is not that I am against peer review per se, but peer review can be administered in hundreds of different ways and it is unlikely that these would be equally robust. For example, even citation metrics are one form of peer-review, actually it is peer review open to the whole scientific community, not just a select crew of study section members. I believe it needs to be optimized and evaluated better. To my knowledge there is no randomized, experimental evidence that peer review (any form of peer review) is better to alternatives (any alternatives). In theory, it is possible that different forms of review or selection criteria may be indicated depending on what outcomes you want to achieve. For example, out-of-the-box, high-risk, bold innovation is unlikely to survive the current review system. Conversely, for more routine research that is useful to do, but anyone could do it, I really see no reason why this should be allocated to single PIs in siloed teams and judged by study sections. This type of work would probably benefit more from team science where any qualifying investigator could equally participate and there would be full transparency and shared effort across the wider scientific community. Currently, applying PIs try to prove that what they do is unique, significant, innovative, and saving the world, when none of that is really true: the work is typically incremental (but not necessarily worthless), it can be done by hundreds if not thousands of others, and it would have been better to do it in an open collaborative scientific community. I think we should try to test rigorously alternative selection approaches. If the current system turns out to be the best, despite its deficiencies, so be it, but I would be greatly surprised if we cannot do any better.

  8. Jeremy Berg Says:

    Josh: Thanks for your comments. The present funding situation (with historically low success rates) is a consequence of the fact that the NIH budget has been essentially flat (and well below the rate of inflation) for nearly a decade after it had been nearly doubled over a five-year period. The “doubling” was intended to (and did) increase the size of the biomedical research enterprise including additional research space, faculty, and trainees. Unfortunately, the flat budgets for a decade have been “undoubling” the NIH budget while the number of investigators competing for resources as continued to grow. The system is now in the process of “re-equilibrating” and many of us are working to guide this process in sensible directions as much as possible.

    Having a career engaged in scientific research (in academia or elsewhere) can be a fantastically satisfying experience. But it is entirely reasonable to be concerned about how such careers can be blended with family and other aspects of life. This is likely always to be a challenge, but it can be made to work.

    I would not worry about any criticisms of NIH hurting your chances of being funded. In my experience, such issues do not arise to any significant extent even for outspoken critics.

    I agree with you that NIH should use a range of different programs for funding. The NIH Director’s New Innovators program uses reviewers who are only loosely matched to the content of the proposal and are encouraged to look for bold and innovative rather than incremental and safe ideas. When I helped establish and run this program, I heard from a number of applicants how much they enjoyed writing proposals about what they really were passionate about as opposed to what they thought was mostly likely to get funded.

    While I understand your logic, I do not think it is wise to try to tailor your application to specific study section members. From my observations, it is hard to predict who on a study section will be most enthusiastic about a proposal. Sometimes the reviewers who are closest to a problem are the most critical whereas other times they are most supportive. It is better to follow your passions and write about what you think is most interesting. This will come through in the proposal. You should also find local (or not so local) critics who will help you prepare the strongest possible proposal.

    Again, that for your comments and best of luck as your career moves forward.

  9. Alan Mayer Says:

    All- I am an HHMI training grant recipient. K08 awardee, R21 awardee, MDPhD now in private practice. I obtained my faculty position in 2003, and after many failed and eventually desperate attempts to get R01 funding, lost my lab in 2009 in the wake of the financial crisis. My department could not honor it’s committment to provide retained startup funds due to steep declines in clinical revenue. Society invested hundreds of thousands of dollars to train me to do science, then a few scared and irresponsible administrators cut the cord when times got tough. I’m sure my story resonates with the 82% of grant applicants who can’t fund their work, and face losing decades of investment in tool development, training techs, etc.
    What a collosal waste this system creates.
    Alan Mayer, MD, PhD

  10. Kenton Sanders Says:

    The main problem at the NIH is the so-called peer review system. The Review Branch and the Study Section mechanism of review is antiquated and greatly flawed. WIth priority scores reduced to historical levels, it is a totally failed concept that is hurting science.
    The NIH has tried to right historical social wrongs with the Study Section mechanism and put very weak and inexperienced, if not novice scientists, on review groups to make important decisions about the direction of science. Search the publications and citations of the publications of the people reviewing your proposals! The NIH peer review system was started by employing people with deep perspective on the field and now often relies upon opinions from inexperienced investigators because they are members of a gender or an under represented minority. Social issues need to be improved, but this is a separate problem from deciding the best science.
    Study Sections are ingrown. People leaving study sections are asked about replacements. They seldom suggest their opponents.
    We need an objective system in which grants are reviewed at large. The NSF uses such as system. Grants should be reviewed by ALL ACTIVE GRANTEES, not a select, politically correct, often inexperienced, and insular committee that may or may not even be current on the hypotheses presented. Why not require every funded investigator to review a limited number of grants per year? Abolish the NIH review branch and pick reviewers based on key words and concepts (easily matched by computer – hire Google as the review branch). This would end the ‘buddy system’ of Study Sections and grant assignments. With open reviews every grant could receive multiple anonymous reviews and priority numbers could actually obtain statistical significance. Reviews should all be done online like reviews of manuscripts to save the time and money of traveling, and eating and staying in hotels for Study Section meetings. Blog if you have questions about a grant, but blog anonymously. Let’s put our money into novel investigation and not into fatuous administration!
    Finally, the NIH intramural program needs to compete on a level playing filed with extramural investigators. Money is too precious now to retain the old system where the intramural program is not competing with the greater family of science. Let the best ideas and scientific productivity prevail.

Comment


Please read our comments policy before posting

Stanford Medicine Resources: