Should you revise and resubmit a triaged NIH Grant application?
January 6, 2014
A December 18 post on the Rock Talk blog issued an update on the funding rate situation for grant applications submitted to the NIH. The data provide
…an early snapshot on success rates for 2013 competing research project grant (RPG) applications and awards.
We received 49,581 competing RPG applications at NIH in fiscal year 2013, slightly declining compared to last year (51,313 applications in FY2012).
… In FY2013 we made 8,310 competing RPG awards, 722 fewer than in FY2012. This puts the overall research project grant (RPG) success rate at 16.8%, a decline from the 17.6% reported in FY2012. One might have expected a bigger drop in the success rates since we made about 8% fewer competing awards this year, but the reduction in the number of applications explains part of it.
emphasis added, as if I need to do so.
See this graph for a recent historical trend on success rates and application submission numbers. With respect to the latter, you can see that the small decrease to 49,581 is not hugely significant. We’ll have to wait for a few more years to be convinced of any trend. Success rates are at an all-time low. This is rather unsurprising to anyone of you that has been paying attention to doing at the NIH and is a result of the long trend toward Defunding the NIH.
Of greater interest in the Rock Talk post was a comment made in response to a query about the fate of initially-triaged applications. A Deborah Frank wrote:
A few months ago, I emailed Rock Talk to ask the same question as Mr. Doherty’s question #3. My query was routed to the Freedom of Information Act Office, and a few months later I received a table of data covering A0 R01s received between FY 2010 and FY2012 (ARRA funds and solicited applications were excluded). Overall at NIH, 2.3% of new R01s that were “not scored” as A0s were funded as A1s (range at different ICs was 0.0% to 8.4%), and 8.7% of renewals that were unscored as A0s were funded as A1s (range 0.0% to 25.7%). These data have at least two limitations. First, funding decisions made in 2013 were not included, so the actual success rates are likely a bit higher. Second, the table does not indicate how many of the unscored A0s were resubmitted.
The NIH data miner / blog team then posted a link to an excel spreadsheet with the relevant numbers for ICs, divided by Type 1 (new) and Type 2 (renewal, aka competing continuation) applications. The spreadsheet notes that this analysis is for unsolicited (i.e., non-RFA) applications and that since the FY2013 funding data were not complete when these were generated (7/15/2013), it is possible that some A0 submitted in this interval may still be funded.
Now, this is not precisely the same as the usual success rate numbers because of
- the aforementioned exclusions
- the way A0 and A1 submitted in the same FY are counted as one application in success rate calculation
- the fact that if an A1 is not submitted it isn’t (cannot be) counted in success rate
Nevertheless, keeping these details in mind it is hard to escape noticing that one is facing steep odds to get a triaged A0 Type 1 proposal funded. On the face of it, anyway. And I have to tell you, Dear Reader, that this is consistent with my personal experience. I can’t recall ever getting a triaged application to the funded level on the next submission. In fact I’m hard pressed to recall getting a triaged A0 funded as an A2 when that was still possible.
Yet I continue to revise them. Not entirely sure why, looking at these data.
Moving along, it is really disappointing that the NIH didn’t go ahead and put all the relevant numbers in their spreadsheet. The thing that PIs really want to know is still terribly obscured by this selected analysis. NIDA, for example, lists 394 unscored Type 1 applications of which 11 (2.8%) were eventually funded. But unlike the now-disappeared CSR FY2004 databook analysis (see here, here for reference to it), they have failed to provide the number of applications that were initially triaged that the PI actually resubmitted as A1! If only half of the triaged applications were amended and resubmitted, then the odds go to 5.6%.
Is this difference relevant to PI decision making? I don’t know for sure but I suspect it would be. It is also relevant to understanding the different success rate for initially-triaged Type 1 and Type 2 applications. The mean and selected ICs I checked tell the same tale, i.e., that Type 2 apps have a much better shot at getting funded after triage on the A0. NIDA is actually pretty extreme from what I can tell- 2.8% versus 15.2%. So if there is a difference in the A1 resubmission rate for Type 1 and Type 2 (and I bet Type 2 apps that get triaged on A0 are much more likely to be amended and resubmitted) apps, the above analysis doesn’t move the relative disadvantage around all that much. However for NIAAA the Type 1 and Type 2 numbers are closer- 4.7% versus 9.8%. So for NIAAA supplicants, a halving of the resubmission rate for Type 1 might bring the odds for Type 1 and Type 2 much closer.
Do these data change my approach? They probably should. However, there is a factor of submission dates here. For any given round, new applications are submitted one month and then amended applications are due the next month. So if you are a few weeks away from the second deadline and considering whether to resubmit an application or not….there is no “new” application that you could submit right now. You have to wait for the next round. So if you are feeling grant pressure..what else are you going to do? Take the low odds or take the guarantee of zero odds?
Final note. I continue to believe, until NIH demonstrates my error very clearly, that considerable numbers of “A0” submissions are really a reworking of ideas that have been previously reviewed. I also believe that these “A0” submissions are disproportionally likely to be funded due to the prior submission/review rounds. Whether this is due to improved grant crafting, additional preliminary data, better approaches, gradual convincing of a study section or Program is not critical here, I’d say all these contribute. If I am correct, then there is value in continuing to work the steps by resubmitting a triaged A0.
__
Additional Reading:
NIH Historical Success Rates Explain Current Attitudes
More data on historical success rates for NIH grants
Old Boys’ Network Favors Men’s Continuing Grants?
DrugMonkey 
January 6, 2014 at 9:21 am
So bottom line, don’t bother resubmitting a triaged A0. Got it.
LikeLike
January 6, 2014 at 9:27 am
I would very much like to see this broken out even further to address what a NI/ESI should do in this setting. I talked recently to my program officer who indicator that in her portfolio, half of funded ESI’s were triaged the first time. Offers some hope, but there are still likely a lot of triaged ESI’s that are never funded or not funded at resubmission.
LikeLike
January 6, 2014 at 9:29 am
My bottom line – resubmit my 26% A1 as an A0. Still trying to figure out how exactly to do that by Feb 5th…
I just had an unscored R21, will have to parse the posted data to see if R21’s fared differently. Probably not!
One really needs a thick skin to play this game in order not to take these rejections personally.
LikeLike
January 6, 2014 at 9:40 am
Why, do you think, do A1’s that went unscored at the A0 stage do so poorly? Is the fact that it was triaged a black mark that cannot be easily overlooked? Is it that the same reviewers get the application again, and not liking it the first time, don’t like it again? Are most of the people that write grants that get triaged likely to make certain grantsmanship mistakes that lead them to write bad A1’s?
I have recently seen grants from really good scientists (who have been well-funded in the past) go unscored. This observation makes me think that if the pink sheets give you good options for revision (clarifying a missed point, adding prelim data that you have or can easily get, etc.), then you should revise and resubmit. The numbers in this post make me wonder if one would be better off resubmitting the application as an A0 to avoid a black mark and maybe get different reviewers and then have 2 more shots at funding.
LikeLike
January 6, 2014 at 9:57 am
This is sort of what I try to explain to some of my younger PI’s but now I have numbers to back it up. Thanks! In my experience, it is rare a triaged application will be funded on resubmission. We have had a couple over the years, but I think the safer route is to scrap the application and start from scratch. Even when the criterion scores don’t look too bad – if you get a feeling that the reviewers shrugged when finishing the application, the chances are the application is not going anywhere. The experienced PI’s have no trouble with this, but I do have to wrestle with younger PI’s regarding this point!
LikeLike
January 6, 2014 at 9:58 am
…..my program officer who indicator that in her portfolio, half of funded ESI’s were triaged the first time
Hmmmm, that sounds…..not quite right. Does she mean that of all ESIs that eventually got a grant funded, half of them had their first ever application triaged? Doesn’t necessarily mean the same app.
LikeLike
January 6, 2014 at 10:17 am
Why, do you think, do A1’s that went unscored at the A0 stage do so poorly?
I suspect a large part is that there are other revised apps coming in that were scored in the prior round. While we still have this tendency for reviewers to at least start from the prior score, this gives the previously triaged A1 a hole to climb out of.
Is the fact that it was triaged a black mark that cannot be easily overlooked?
I am not certain the mere fact of triage (vs, say, a 50%ile score) is the problem, although perhaps there are still some reviewers that have a huge mental block on this.
Is it that the same reviewers get the application again, and not liking it the first time, don’t like it again?
The review process doesn’t do very well at communicating “this is so boring/meaningless that it doesn’t matter what you do, it ain’t getting a good score ever”. Some think they can glean this from comments (hi PP!) but I’m less certain. Also, it may be the case that more of the section members than just the current three will find it boring. So new reviewers is no guarantee.
Are most of the people that write grants that get triaged likely to make certain grantsmanship mistakes that lead them to write bad A1’s?
I think you have this confused a bit. sure there are going to be bad grant writers that struggle with every aspect, including revision in face of critique. but having written a lot of triaged apps and having been given scores that are probably as good as it got in a given study section round…I am very much not convinced that grantsmanship is the main driver. This is buttressed by my years on study section and other happenstance views of the proposals of highly-funded luminaries…they write decent grants, sure, but not so awesome as to be a clear difference in grantsmithing compared to us plebes that fight for every one. The relative triage protection of the biggest PIs over the past decade and a half has more to do with their track record than their grant crafting.
LikeLike
January 6, 2014 at 11:09 am
“The relative triage protection of the biggest PIs over the past decade and a half has more to do with their track record than their grant crafting”.
I totally share this view.
LikeLike
January 6, 2014 at 1:13 pm
There ought to be a more official distinction with regard to reasons for triage, such that if an app is walloped heavily on Significance, the proposal is just plain blocked from resubmission period.
Alternatively, based on these data, just make it official and block all triaged applications from subsequent resubmission. Maybe then use the extra reviewer time and money to bring back the A2, perhaps restricting it to A1 proposals that come in under ~30%ile or something.
Hell, I think any proposal that consistently scores better than 20%ile should be allowed to be resubmitted ad infinitum until it gets funded. Having to completely restructure a proposal because it couldn’t quite make the last yard over what is accepted to be a rather arbitrary pay-line is insane.
LikeLike
January 6, 2014 at 1:14 pm
“The relative triage protection of the biggest PIs over the past decade and a half has more to do with their track record than their grant crafting.”
Very true…and the real problem with this is the circular logic involved in maintaining said track record.
LikeLike
January 6, 2014 at 1:31 pm
Joe says,
“The numbers in this post make me wonder if one would be better off resubmitting the application as an A0 to avoid a black mark and maybe get different reviewers and then have 2 more shots at funding.”
Fine and dandy, but if they (NIH’s nitpickers) come to the conclusion that your “A0” is really an “A1”, this strategy won’t work – NIH will either reject the application or assign it the A1 designation (not sure which). To avoid that outcome, you’d have to make your new A0 significantly different from the triaged A0.
LikeLike
January 7, 2014 at 5:24 am
I think what you said at the end is the most meaningful: if you don’t resubmit your chances are zero. That’s my calculation anyway.
FWIW I had an RO1 funded on an A2 that got triaged as an A0 and an A1.
LikeLike
January 7, 2014 at 5:43 am
It is my understanding that a change in mechanism counts as a new grant even if the studies are identical. This may not be helpful for some applications or needs. As an example, I recently changed a triaged 2 year R21 to a 3 year R01. The A0 of the Ro1 got scored. Waiting on Feb/Mar study section for A1 outcome. I know of several folks that have pealed out an Aim from an triaged R01 that had enthusiastic support and converted that Aim to an R21–with some success.
LikeLike
January 7, 2014 at 8:56 am
PR-
You are underlining my point that many “A0” proposals that do well were previously reviewed in some other guise. NIH has to know this but keeps right on pretending than they have shortened the time to funding a proposed idea.
It KILLS me. They need to fix study section proclivity for the air traffic pattern delay.
LikeLike
January 8, 2014 at 9:16 am
“For any given round, new applications are submitted one month and then amended applications are due the next month. … what else are you going to do? Take the low odds or take the guarantee of zero odds?”
DM you have expressed this view before, but I think there is a fallacy here, at least as I see it applied to myself, because my time is both limited and fungible. Experience tells me that substantially revising and resubmitting an R01 is going to cost me ~50 hours, and that will be 50 hours that I am guaranteed not to be working on any of the five manuscripts on my desk waiting only to be written, or doing the research and planning to set new studies in motion, or spending time with my kids that are still young enough to want to spend time with me. Each of these alternative uses of my 50 hours comes with its own cost-benefit analysis, and there are strong arguments in favor of each. I have not previously thought the expected value of resubmitting a triaged application to be worth my time, and the data you are showing here only confirm my thinking. It is nice though to see the real data on the funding success of these applications, though!
LikeLike
January 8, 2014 at 10:41 am
First off, thanks DM for the analysis and stats. I agree that grants that were not funded (triaged or not) are making re-appearances as a new A0.
Congrats Namnezia on getting funded after triage – I never was able to climb out of the hole (even with an A2).
Meshugena313 in re: thick skin. Indeed. One of the hardest lessons to learn. But I think the more important and difficult lesson is the one that Crystaldoc mentions: the tradeoff between time on grants vs. time on mss, especially for ESI. If you don’t have the pubs to back up your productivity, experience and feasibility, there is no point in writing the grant.
I agree that the holding pattern (we’ll get to it next time) – either implicitly or explicitly has a lot to do with the problems.
LikeLike
January 8, 2014 at 11:28 am
Good point CD! Manuscripts have to keep going out, for sure.
LikeLike
January 8, 2014 at 5:01 pm
I’ve always felt it is better to go with the Devil you know (the reviewers that reviewed the A0) and try to placate them then the Devil you don’t know (a new set of reviewers and comments). If an application is triaged I throw out the aim that garnered the least enthusiasm and add a new aim supported with new preliminary data (since the data presented in the A0 has hopefully been published). In other words, I revise the application. Window dressing does no good for a triaged application. I like that idea that somebody mentioned previously about a “do not resubmit box” — that would certainly take all doubt out of the decision.
LikeLike
January 9, 2014 at 4:51 am
I have seen many triaged grants go from triage to funded. On one study section, I saw a grant go from triaged to top grant in the next cycle of the study section. You’ve already worked hard on a triaged grant, turn it around and fix it, if you can.
It’s the “if you can” that matters. Just as with a paper “rejection”, you need to read the reviews. Some rejections are actually “revise and resubmit”, while other rejections are “don’t waste your time”.
As to DM’s issue about air traffic control, DM seems to have the opinion that there are two options: “waiting in line” and “getting good grants funded early”. But I claim that there are two options: “waiting in line” and “playing the lottery”. Because there is so much noise in the fundability of grants (there is a difference between top 10% [funded] and triaged, but little to no difference between top 10% [funded] and 20% [not funded]), the actual funding ends up being a lottery. There was an advantage to the air traffic control system – you knew you were in line (or had a good chance to be). What we really need is a system where you know you have a good shot at continuing getting funded because you’ve been making good progress previously.
I’ve suggested this before, and will suggest it again: 1. there is a high bar to get into the system, based on the proposed work-to-do, 2. once you are in the system, renewing a grant is based on productivity, with very clear expectations. Each PI would be allowed one grant in this easily-renewable category. Specifically, there are two kinds of R01s – a “primary” and “secondary+”. To get the initial primary R01, the PI has to go through a full study-section-like nasty review process. To renew the R01, the review process is entirely “was good work done in the last cycle”, and people should know that they’re either “doing fine” or “in trouble”. If you want more than one R01, you have to fight in the whole nasty review process for additional ones. If you lose your primary R01 (because you screwed up a cycle), you have to go through the high bar to get back.
This would do three things: (1) it would remove the waiting-in-line issue that DM hates (as do I), (2) it would reduce both the grant-writing burden (if you just want your one R01, you don’t need to write much) and the reviewing burden (because we wouldn’t be wasting time reviewing all those extra grants), and (3) it would give you the ability to plan.
I should point out, this is very much what my older colleagues tell me was the system in their time. Renewing a grant was easy if you had made progress. Lots of people spent entire careers on a single R01.
LikeLike
January 9, 2014 at 4:52 am
PS. @GAATTC – There is officially a “do not resubmit” category in study section review at NIH. Something like “not suitable for reconsideration” or the like. But I’ve never seen it used.
LikeLike
January 9, 2014 at 4:54 am
PPS. In my suggestion, secondary R01s would not be renewable. If you want to continue the work on the secondary R01, you would need to fight in the nasty study section each cycle.
LikeLike
January 9, 2014 at 5:55 am
My first RO1 went from triaged on the A0 to funded on the A1, so it is possible.
On the other hand, the biggest difference between to A0 and A1 was that my first independent papers as the PI of a new lab had appeared. So maybe it was just a case of “show us that you can actually set up a functioning lab”.
LikeLike
January 9, 2014 at 5:51 pm
AcademicLurker,
Not sure with other ICs, at least NINDS are funding ESI up to 25%tile. I assume you were a new investigator/ESI. Do you mind telling us the %tile or the impact score of your A0 and Α1?
LikeLike
January 10, 2014 at 6:52 am
The A1 was 15%. I still benefited from HLB’s new investigator policy. The payline that year was 12% but they boosted all new investigator scores by 10 points, I believe.
LikeLike
January 10, 2014 at 7:32 am
It also depends on the way it is triaged. If it is triaged because of a consistent poor score across the board, it is probably not going to do well in revision. If it is triaged because you have a split between reviewers who gave it all 1s and a reviewer who completely trashed it, then the flaw could be in the proposed work (high numbers are right; trash it) or in the writing (you didn’t communicate the significance to that reviewer; it can be fixed).
LikeLike
January 10, 2014 at 7:50 am
But qaz- this is “established investigator” you speaking. What would noob-you have thought when facing a tiny slice of the pie and a raft of plodding grey beards sucking up 85% of the grant funding? Based merely on demographic advantage and not merit. Think through the implications of your plan.
LikeLike
January 10, 2014 at 6:51 pm
DM – Are you kidding? How is that different from what we have now? Lots of graybeards sucking up 85% of the grant funding?
It’s certainly what I faced on entering the system. At the time, the only help new investigators got was that program would occasionally (very occasionally) help a new investigator with a special pedigree by reaching down just below the payline. New investigators were reviewed in the same mix with 20-year veterans. Now, there’s K99/R00’s and DP2’s and ESI and NI help and lots of other special stuff.
I would argue that my suggestion would actually help the noobs because most of those greybeards would not be fighting for the noob money, they would have their steady R01. If we are really concerned about this, we could have separate competitions for getting-a-primary R01 (that first high-bar-entry) and for secondary-R01s. My proposal would also (I argue) reduce the number of extraneous second R01s, because people could survive on one R01, which should free up some money. (Not much, I know.)
There’s no question that we would need to have the right amounts of money divided among the different groups – this much for established investigators, this much for new slots entering, etc. What NIH needs to do is to treat funding sets like a diversified portfolio – this percentage assigned to young, risky, and growing faculty, this percentage assigned to continuing, ongoing, stable faculty, this percentage assigned to special extra R01s, etc. Everyone knows that the easiest way to lose your shirt in the stock market is to chase the “best stocks” by trading constantly. There’s no reason to have the noobs fighting with the returning established faculty.
LikeLike
January 10, 2014 at 9:07 pm
Are you fine with the notion that your stable grant might be $250K whereas the next person might get $750K as her baseline grant?
LikeLike
January 11, 2014 at 6:01 am
No, I’m not. I agree that that’s a difficult point. And it might be a reason not to implement my stable-R01 idea, because we might find ourselves in the same boat as indirect costs where certain universities, fields, or individuals negotiate themselves sweet deals that make their budgets inordinately expensive. (*)
But I’ve reviewed LOTS of grants for lots of places in a variety of fields (including clinical, as well as NSF, NIH, and NASA) and I’m pretty convinced at this point that there is absolutely no reason for an R01 to be $750K without bloat.
There are grants that are not basic science grants that might need that much money, but then we are no longer in the equivalent set of goals. I think it’s a major mistake to include a basic science grant in the same category as a Phase X multi-site clinical study. Which again speaks to the portfolio point.
* Actually, a much simpler and smaller proposal would be to allow study section to take budget into account when scoring a grant, so that we were scoring based on “is the impact of science worth this expense” rather than scoring based on “impact of science” and then deciding whether the budget is too high. I’ve seen lots of grants where the cost is what the grant would cost (particularly at that university in that city) but that we could do two grants (in another field at another university in another city) for the same price. So we can’t say “the budget is too high”, but we would say “that impact isn’t worth the price”.
LikeLike
January 11, 2014 at 12:14 pm
What a crock, qaz. In other words, your “basic science” at wherever you are is relatively inexpensive so everyone else should have to go through extraordinary justification. But not you, you should get your default basal amount with minimal competition.
Sorry, try again.
LikeLike
January 11, 2014 at 12:17 pm
Should study sections also be provided a list of the funded grants and PIs already doing a certain technique/model/what all and make sure to do ongoing balancing of the NIH for it?
LikeLike
January 13, 2014 at 6:16 am
It also depends on the way it is triaged. If it is triaged because of a consistent poor score across the board, it is probably not going to do well in revision. If it is triaged because you have a split between reviewers who gave it all 1s and a reviewer who completely trashed it, then the flaw could be in the proposed work (high numbers are right; trash it) or in the writing (you didn’t communicate the significance to that reviewer; it can be fixed).
Another key factor is which criterion scores were bad. If the Significance and Innovation were generally excellent, but the Approach was not, then that is a good bet to resubmit. If the Significance and/or Innovation were poor, then that is much harder to overcome.
LikeLike
January 13, 2014 at 1:39 pm
But, if it were about the science, shouldn’t it matter if the work could be done less expensively elsewhere?
Companies are doing this calculus all the time and have effectively gotten states to compete heavily for their work. Boeing is being given tax breaks in Washington, for example, that the federal research funds don’t (the particular bone people pick is on sales tax for equipment, mind you WA doesn’t have a income tax, so they’re not recouping federal funds on salaries like the higher tax states)
If, as federal taxpayers, we think placing work at a certain institution creates benefits for the institution and the region, why isn’t it reasonable to expect the institution and area to invest in the endeavor as well?
LikeLike
January 13, 2014 at 1:57 pm
Nonprofits like Universities and Research Institutes don’t enjoy tax breaks zb? Of course they do.
but yes, you have the first plausible argument I’ve heard on this topic. Kudos.
Question. Is this for all federal contracts for goods and services or just the exceptionally high profile and high budget items? Do all participants in the chain that produces the latest fighter jet participate? Or only the principal contractor?
If it *is* only the biggest companies that get the breaks….does the government expect the smaller ones to cost account with efficiency similar to the larger companies? Or is this just a deal by deal scenario?
I think what we’ll find if we follow your scenario down is that at best it matches the federal indirect cost situation. A situation of high variability in tax payer co-support for the companies providing the government with what it wants. Also a high variability in profit margin. Universities and nonprofit research institutions already come out ahead on the profit issue….
LikeLike
January 14, 2014 at 8:36 pm
My current R01 was triaged and funded on A1. I couldn’t imagine giving up the relatively easy opportunity to respond to critiques and resubmit. What is there to lose other than your sanity?
LikeLike