Initial outcome of limiting NIH apps to a single revision?

January 20, 2011

The latest Peer Review Notes issue [pdf] from the Center for Scientific Review of the US National Institutes of Health reports some initial data on their move to eliminate the second round of revision of grant applications.
Personally, I thought this was very likely only a partial fix to the problem. As I’ve discussed, I was no fan of the way that available rounds of revision led study sections to refuse to get serious about reviewing apps until they had returned from at least one round of review. So I think it is a good idea to try to break this particular time-wasting bit of study section culture.

However I looked back to the prior event (in 1996) when revisions were limited to two from the previous lack of any limit (-01A6 FTW! w00t Perrin, w00t Croce! Takes fortitude to go through the original plus 6 revisions to finally break through to funding!). In looking at the trends for funded apps that made it to funding unrevised versus on the first or second round of revision, I was unimpressed that the prior limitation to two revisions did anything about the trend for unrevised applications.

An apparent bump in unrevised grants being funded is more likely due to the great NIH Doubling (now UnDoubled) interval, IMO, because the numbers for A1 and A2 revisions kept marching upward year by year. As soon as the budget flattened out, those numbers came back down towards the trendline. As you can see in the following figure, things got even worse for original submissions in the 2006-2008 interval. (N.b. the data in this figure are raw numbers, the data below are percentages.)
The aforementioned issue of the Peer Review Notes has the below figure and the accompanying observations:

About 10 years ago, 60 percent of the R01 applications NIH funded were A0 applications–those that went through the review process just once. In recent years, only 30 percent of all funded R01 applications were A0s. “Applicants with exceptional applications were frustrated.” said CSR Director Dr. Toni Scarpa. “It was like they had to get a ticket and wait in line for a year or two. Reviewers were frustrated as well . . . reviewing the same applications over and over again with little effect on the final results.”
NIH abolished second resubmissions (A2s) of NIH grant applications so it could fund the best applications sooner. Recent data suggest that the policy is working: NIH now funds more A0s.


Now, as the CSR newsletter notes, the ARRA funding probably had “some effect” on the 2009-2010 numbers. I’d argue a big effect, but whatever. The most important thing here is that the A2s are clearing the system. There weren’t as many being submitted, particularly for the 2010 FY of funding. So they are dropping out of the pool of funded applications. But notice that the A1 application numbers aren’t budging? Or at least not by much. Yet? One hopes. But if this number doesn’t budge, then I am very much less impressed that CSR has fixed the problem.
I think this shows that study sections are still nitpicking the original submissions and putting them into the queue instead of doing what I think they should (i.e., taking a “fish-or-cut-bait” approach of looking for the broad strokes and ignoring minor violations that basically amount to grantsmithing issues, empirical predictions, etc). If this had been fixed, the relative proportion of A1s should be dropping just like the A2s….maybe with a slightly shallower trendline, but dropping.
Of course I also want better information on the even bigger issue here. I assume that PIs who get a halfway decent, but not fundable, score on their A1 applications are going to turn that baby right around as a new submission. In fact I recommend strongly that all of my readers do exactly this. CSR has made a lot of noise about how they are going to stringently weed out thinly disguised A2s coming in as new submission but I doubt whether they can do this on a consistent basis.
We applicants are quite clever when put in a corner, you know.
So there are going to be “original” submissions coming in that have the benefit of most of the ideas having been previously reviewed. These do not count if the NIH is going to brag on making the system more efficient!
I’d like to see them doing some anonymous exit surveying of study sections in which they asked the reviewers to report whether they evaluated any “original” submissions that were actually disguised A2s of applications previously reviewed in that section.

27 Responses to “Initial outcome of limiting NIH apps to a single revision?”

  1. Pascale Says:

    Unfortunately, I have had colleagues get rejected for their “new proposal” being too similar to the last revision. And, frankly, there are only so many ways to look at a problem or phrase it.
    IMO, study sections do too much nit-picking / micromanaging, looking for any reason to ding a proposal in this time of scarce resources.


  2. Do you think you’ll see the success rate for A0’s plummet when all those folks who had challenge grants start submitting new R01s, and will this influx maybe bring us right back to allowing a A2 revisions?


  3. DrugMonkey Says:

    I doubt the NIH is going to go back to A2 revisions any time soon. Too big a move for them to abandon it quickly.
    What I think more likely is more concerted effort to disguise the same old stuff in new guise in a way that gets it past the smell test. Might be only changed Aims, might be only enough change in focus to get it into another study section. Perhaps we’ll see some mechanism-jumping and a whole lot of R01s being broken down into R21s and R03s?


  4. DrugMonkey Says:

    Pascale, do you happen to know if these applications got caught by CSR or only after going to reviewers? and did you get a chance to see for yourself how similar the grant apps were?


  5. I don’t understand all of your nitpicking about whether or not the elimination of A2s is going to make the process more efficient. In 2008, it looks like 30, 38, and 32% of A0, A1, and A2s made up the overall awards, respectively. As A2s get fazed out, their 32% of the pie will get disbursed to the A0s and A1s.
    Even if A1 percentages never go down from 38%, that would mean that 62% of awards are A0 submissions. This, to me, does not sound like a problem.
    IMO, anyway you look at this program, it will be a success, because the likely worst case scenario is 30% A0 and 70% A1 apps being funded, which means 32% of awards are being awarded one cycle earlier.


  6. whimple Says:

    Another consideration is that the NIH is going to be pressured to increase the ESI bump-up. For an ESI, not getting the A1 funded combined with stringent re-application pruning is a major disaster not only for the ESI, but also for the Institution that put ~$1M into that person. I predict a lot of nasty churn in the untenured ranks, with unclear long-term consequences.


  7. Anonymous Says:

    The whole NIH granting system has become such a nitpicking ordeal, which now is more of a hindrance to the advancement of scientific research than a stimulant. Ninety percent of the apps are never being funded and the 10% being funded is not necessarily the best scientific projects submitted. The system is broken and the nitpicking changes described here are just that, nitpicking.


  8. DrugMonkey Says:

    This, to me, does not sound like a problem…which means 32% of awards are being awarded one cycle earlier.
    This is the question, Candid. Will these awards really be made “one cycle earlier”? or are you having the same bias for revised grants being papered over by the fact that some fraction of those “A0” apps are in fact disguised A2s.
    The question, in essence, is whether study sections’ tendencies to nitpick over irrelevant grantsmithing issues has been reduced or not. If it has not been reduced, we have the same amount of effort (i.e., number of revised submissions) being papered over with this little trick of NIH’s. If it has been reduced, I argue that we should see this in a reduction of the number of grants being funded on the A1 stage.


  9. DrugMonkey Says:

    Ninety percent of the apps are never being funded
    This is not true. The graph in this post makes the critical point about the eventual fate of applications that did not get funded on the first round. Although those data only go to 2006 the same principle holds. You have to account for the allowable revisions to make your “never” statement. What I’m arguing with this post is that you also have to take into account the thinly disguised A0 resubmissions…although the NIH has basically ensured that there is no way to track this with a high level of accuracy.


  10. DM, I see your point, although I still think there will be at least modest improvements in the time from initial submission to funding.
    It would be useful to get stats for first-time submitters before and after A2s were cut out. You then eliminate the pool of disguised resubmissions.


  11. Anonymous Says:

    While only a sample, the three letters published recently summarize pretty accurately the state of NIH-supported biomedical research in the US.


  12. DrugMonkey Says:

    Anonymous the only line in those three letters that is worth much is this one

    Many think it a good idea as long as the “everyone has enough but no one is huge or overly rich” rubric is only applied to others. Whining for dollars is the #1 academic indoor sport, and no one does it better than biomedical researchers!

    People just love solutions that work out well for them and screw “those other overfunded, underproductive, unimportant scientists from some East Jebus State University I’ve never heard of”. Or similar.
    The comment about the French CNRS system is woefully naive. The funding-for-life (or really any program-based funding scheme) has plenty of drawbacks having to do with incentives to stay active/innovative/vigorous, lack of opportunity for deserving newbs, scientific cronyism, etc. Those who think the current NIH funded system is bad on these factors needs to talk to a young French (or German) scientist about their career issues.


  13. These changes–like almost all of the enhancing peer review changes–are really designed to hasten attrition of substantial numbers of marginal PIs/insitutions that were attracted into the system during the budget doubling of the late 90s/early 00s. These people/institutions need to be flushed from the system so that success rates can normalize.
    Of course, this kind of plan cannot be openly sold to Congress, so it is packaged as “enhancing peer review” and “funding grants sooner”, but it is really all about making it easier for quality investigators to get their grants and more difficult for marginal investigators. If you were the kind of investigator who could never get a grant funded until you had been beaten to an A2 by study section, or who needed 25 full pages of preliminary data to convince a study section to fund your boring/incremental science, now you’re never gonna get a grant and you’re gonna be flushed.


  14. Anonymous Says:

    There is much truth to what’s stated in #13. The ideal funding situation is to flush all institutions from the system. Funding should goes directly and only to the investigator and her project. Institutions and their fat administrative cushion should never be part of the formula for funding scientific research. Universities and research institutions should get their funding from all the other sources from which they used to get their funding. The NIH should never be expected to fund the fat salaries of my Dean and my vice president for research or the new research building that stands empty now, since many of the NIH-supported PIs, who were recruited in the past 10 years, jumped ship to other, more glamorous institutions that can charge the NIH higher percentage of indirect costs, thus having even bigger and fancier research buildings.


  15. DrugMonkey Says:

    This ain’t going to happen my friend. Not until and unless they reconstitute study sections to match your warped idea of quality.


  16. Not until and unless they reconstitute study sections to match your warped idea of quality.

    Dude, what the fucke do you think they are doing with all this “transformative”, “pioneer”, “impact” shitte? It ain’t gonna happen overnight.
    And BTW, this has nothing to do with “my” idea of quality, which I haven’t said jacke fucken dicke about. This is the NIH doing what it perceives as necessary to preserve itself. I’m not making any normative claims here at all, just describing what is happening.


  17. whimple Says:

    This is the NIH doing what it perceives as necessary to preserve itself.
    Really? In what manner is the NIH being threatened?


  18. DrugMonkey Says:

    This is the NIH doing what it perceives as necessary to preserve itself.
    Happy talk about translating science into medicines is consistent with this, granted. Your line that the goal is to shell out all the inferior labs and/or Universities that lucked into funding when the budget doubled, to concentrate the $$ with the elites, is not.
    Reason being…..ding, ding, ding…Congressional Districts. What is necessary for NIH to “preserve” itself is to keep as many CongressCritters as possible thinking that the NIH routes Federal dolla-dolla bills into their home districts.
    This is why I am curious as to how you arrive at your conclusion about the alleged strategy to shell out the weaker, so called “incremental” laboratories.


  19. What is necessary for NIH to “preserve” itself is to keep as many CongressCritters as possible thinking that the NIH routes Federal dolla-dolla bills into their home districts.

    That’s very quaint, but electoral politics in the US doesn’t work this way any more.


  20. DrugMonkey Says:

    I note you have still neglected to explain whereby you arrive at your flamboyant predictions that the NIH is attempting to ashcan those scientist who you deem to be unexciting and pedestrian.


  21. Dude, I’m not “deeming” anybody to be anything, so calm the fucke down and stop junping to conclusions. I’m interpreting the actual behavior of high-level NIH staff and the decisions they have been making over the last five years.
    What do you thinke is the ultimate long-term goal of the “enhancing peer review” initiative to fund the “best” science with the highest “overall impact”? What do you think is the reason for the NCRR shitcanning? What do you think is the purpose and outcome of initiatives like K99/R00, Young Innovator, Director’s Pioneer, CTSA, more generous paylines for ESIs but not NIs, shortened applications, more telegraphic summary statements? How much money actually gets spent on R15s versus CTSAs?
    Are these things designed to expand the scope of investigators and institutions that receive substantial NIH funds? Or are they designed to preserve and reinforce a core of what *NIH itself* deems the “best” science at the “best” institutions, and to hasten attrition of others?


  22. DrugMonkey Says:

    see, now was that so hard?
    Look you are the one that divines a Grande Purpose behind all of this. I believe the null hypothesis has to include random personal agenda following, the usual clusterborkery of hands not talking to each other and a desperate attempt to satisfy Congressional and other non-scientist critics on the topic of MoarMedicines! My suspicious hat might be willing to entertain the idea that there is something more than mere personal agenda, i.e., an attempt to divert yet more public funds into private companies in some way. A distant third or fifth is your contention that there is actually a coherent attempt to remove NIH funding from what you see as lesser institutions and laboratories.


  23. A distant third or fifth is your contention that there is actually a coherent attempt to remove NIH funding from what you see as lesser institutions and laboratories.

    How many fucken times do I have to tell you that this has nothing to do with what *I* see in terms of lesser or greater labs or PIs, and that I have not opined on that issue at all. Stop putting words in my fucken mouth, fuckeasse.


  24. DrugMonkey Says:

    Dude, own it. You make the same sorts of comments enough, over a sustained interval and it is going to start sounding like your opinion.


  25. Dude, don’t be such a fucken dumshitte: what I wrote backe then was also purely descriptive, not normative.
    Anyway, how the fucken fucke did you find that olde shitte? You got some kind of motherfucken dossier on me or something?


  26. pinus Says:

    DM has a dossier on everbody. creepy shit man.


  27. annie Says:

    Did not know either that monkey d had a dossier on everybody. He must have a huge FBI-proof computer.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: