Our longtime blog commenter dsks is always insightful. This time, the proposal is such a doozy that it is worth dragging up as a new post.

… just make it official and block all triaged applications from subsequent resubmission. Maybe then use the extra reviewer time and money to bring back the A2, perhaps restricting it to A1 proposals that come in under ~30%ile or something.

Hell, I think any proposal that consistently scores better than 20%ile should be allowed to be resubmitted ad infinitum until it gets funded. Having to completely restructure a proposal because it couldn’t quite make the last yard over what is accepted to be a rather arbitrary pay-line is insane.

On first blush that first one sounds pretty good. Not so sure about the endless queuing of an above payline, below 20%ile grant, personally. (I mean, isn’t this where Program steps in and just picks it up already?)

This reminds me of something, though. Unlike in times past, the applicant now has some information on just how strong the rejection really was because of the criterion scores. This gives some specific quantification in contrast to only being able to parse the language of the review.

One would hope that there would be some correlation between the criterion scores and the choice of the PI to resubmit. As in, if you get 4s and 5s on Approach or Significance, maybe it is worth it. 7s and 8s mean you really better not bother.

A December 18 post on the Rock Talk blog issued an update on the funding rate situation for grant applications submitted to the NIH. The data provide

…an early snapshot on success rates for 2013 competing research project grant (RPG) applications and awards.

We received 49,581 competing RPG applications at NIH in fiscal year 2013, slightly declining compared to last year (51,313 applications in FY2012).

… In FY2013 we made 8,310 competing RPG awards, 722 fewer than in FY2012. This puts the overall research project grant (RPG) success rate at 16.8%, a decline from the 17.6% reported in FY2012. One might have expected a bigger drop in the success rates since we made about 8% fewer competing awards this year, but the reduction in the number of applications explains part of it.

emphasis added, as if I need to do so.

See this graph for a recent historical trend on success rates and application submission numbers. With respect to the latter, you can see that the small decrease to 49,581 is not hugely significant. We’ll have to wait for a few more years to be convinced of any trend. Success rates are at an all-time low. This is rather unsurprising to anyone of you that has been paying attention to doing at the NIH and is a result of the long trend toward Defunding the NIH.

Of greater interest in the Rock Talk post was a comment made in response to a query about the fate of initially-triaged applications. A Deborah Frank wrote:

A few months ago, I emailed Rock Talk to ask the same question as Mr. Doherty’s question #3. My query was routed to the Freedom of Information Act Office, and a few months later I received a table of data covering A0 R01s received between FY 2010 and FY2012 (ARRA funds and solicited applications were excluded). Overall at NIH, 2.3% of new R01s that were “not scored” as A0s were funded as A1s (range at different ICs was 0.0% to 8.4%), and 8.7% of renewals that were unscored as A0s were funded as A1s (range 0.0% to 25.7%). These data have at least two limitations. First, funding decisions made in 2013 were not included, so the actual success rates are likely a bit higher. Second, the table does not indicate how many of the unscored A0s were resubmitted.

The NIH data miner / blog team then posted a link to an excel spreadsheet with the relevant numbers for ICs, divided by Type 1 (new) and Type 2 (renewal, aka competing continuation) applications. The spreadsheet notes that this analysis is for unsolicited (i.e., non-RFA) applications and that since the FY2013 funding data were not complete when these were generated (7/15/2013), it is possible that some A0 submitted in this interval may still be funded.

Now, this is not precisely the same as the usual success rate numbers because of

  • the aforementioned exclusions
  • the way A0 and A1 submitted in the same FY are counted as one application in success rate calculation
  • the fact that if an A1 is not submitted it isn’t (cannot be) counted in success rate

Nevertheless, keeping these details in mind it is hard to escape noticing that one is facing steep odds to get a triaged A0 Type 1 proposal funded. On the face of it, anyway. And I have to tell you, Dear Reader, that this is consistent with my personal experience. I can’t recall ever getting a triaged application to the funded level on the next submission. In fact I’m hard pressed to recall getting a triaged A0 funded as an A2 when that was still possible.

Yet I continue to revise them. Not entirely sure why, looking at these data.

Moving along, it is really disappointing that the NIH didn’t go ahead and put all the relevant numbers in their spreadsheet. The thing that PIs really want to know is still terribly obscured by this selected analysis. NIDA, for example, lists 394 unscored Type 1 applications of which 11 (2.8%) were eventually funded. But unlike the now-disappeared CSR FY2004 databook analysis (see here, here for reference to it), they have failed to provide the number of applications that were initially triaged that the PI actually resubmitted as A1! If only half of the triaged applications were amended and resubmitted, then the odds go to 5.6%.

Is this difference relevant to PI decision making? I don’t know for sure but I suspect it would be. It is also relevant to understanding the different success rate for initially-triaged Type 1 and Type 2 applications. The mean and selected ICs I checked tell the same tale, i.e., that Type 2 apps have a much better shot at getting funded after triage on the A0. NIDA is actually pretty extreme from what I can tell- 2.8% versus 15.2%. So if there is a difference in the A1 resubmission rate for Type 1 and Type 2 (and I bet Type 2 apps that get triaged on A0 are much more likely to be amended and resubmitted) apps, the above analysis doesn’t move the relative disadvantage around all that much. However for NIAAA the Type 1 and Type 2 numbers are closer- 4.7% versus 9.8%. So for NIAAA supplicants, a halving of the resubmission rate for Type 1 might bring the odds for Type 1 and Type 2 much closer.

Do these data change my approach? They probably should. However, there is a factor of submission dates here. For any given round, new applications are submitted one month and then amended applications are due the next month. So if you are a few weeks away from the second deadline and considering whether to resubmit an application or not….there is no “new” application that you could submit right now. You have to wait for the next round. So if you are feeling grant pressure..what else are you going to do? Take the low odds or take the guarantee of zero odds?

Final note. I continue to believe, until NIH demonstrates my error very clearly, that considerable numbers of “A0” submissions are really a reworking of ideas that have been previously reviewed. I also believe that these “A0” submissions are disproportionally likely to be funded due to the prior submission/review rounds. Whether this is due to improved grant crafting, additional preliminary data, better approaches, gradual convincing of a study section or Program is not critical here, I’d say all these contribute. If I am correct, then there is value in continuing to work the steps by resubmitting a triaged A0.
Additional Reading:

NIH Historical Success Rates Explain Current Attitudes

More data on historical success rates for NIH grants

Old Boys’ Network Favors Men’s Continuing Grants?