On resubmitting unfunded A1 NIH grant applications
April 8, 2014
Well, well, well.
The NIH limited applicants to a single revision (“amendment”, hence -01A1 version) of an unfunded “new” grant submission (the -01 version, sometimes called “A0”) in 2009.
This followed the action in 1997 to limit revisions to two (see RockTalk chart), which hurt PIs like Croce and Perrin. (Six revision? Wow, that is some serious persistence guys, my hat is off.)
I wasn’t really paying attention to such matters in 1997 but there was some screaming in 2009, let me tell you.
Delusional Biomedical Researchers Seek Repeal Of Arithmetic
More on the new NIH policy on grant application revisions
Initial outcome of limiting NIH apps to a single revision?
NIH re-evaluating ‘two strikes’ rule – Updated
Crocodile tears from experienced NIH investigators over the discontinued A2 revision
I don’t know how many people actually got stuck in the filter for submitting a A0 that was too similar to their prior, unfunded A1. I heard of a few, so it did happen. On the flip side of that, I’ve sure as heck been putting in more than two versions of a proposal which is designed to fund the same area of interest in my laboratory. I have not yet been flagged for it. My initial reaction that any PI who has an ounce of creativity ought to be able to come up with a credible alternative take on their project is still my current take.
Nevertheless, rumor has it that changes are in the wind.
Pinko Punko made an interesting comment on the blog:
DM, I heard the craziest thing today- the possibility of removing the “substantial revision” criterion for new A0 related to previous A1. Supposedly announcement soon- I was kind of surprised.
This was news to me but I have heard things from about five independent sources in the past few days that tend to confirm that changes are being considered.
The most consistent rumor is that new grants will no longer be checked for similarity to prior unfunded proposals. They will be called new grants, but there is no apparent reason for this. In all ways I can see, this is going to be a return to the days prior to 1997 where you could just endlessly revise until the study section relented.
The supposed benefit of reduced “time to award from original proposal” is now going totally by the wayside. I mean, the NIH will still be able to lie and say “look it was an A0!” if they want to but this is even less credible.
More dangerously, the will of study sections to endlessly queue applications will be released from whatever tepid effect the A1 limit has produced.
This is a very BadThing.
__
whoa. I found three A7 projects. All three are competing continuations. I can’t EVEN….five and six year apparent funding gaps for two of them. The other I can’t work out why there is no apparent gap in funding.
DrugMonkey 
April 8, 2014 at 4:36 pm
I wonder if that means they (the NIH) will get rid of the introduction section when you resubmit since all applications may be considered new? I’m not sure responses to previous critiques helped very much anyway…
LikeLike
April 8, 2014 at 5:26 pm
I’m not necessarily convinced it is bad. Given data that there is a large amount of noise in evaluation, fresh look can be OK. I think the presumption that somebody putting a 7 or 8 grant in over and over is less of a worry, there is no way they would last that long. The half life of their lab would be shorter than the time to fund that. I wonder if it will just lead to an expectation of massive amounts of preliminary data, because some proposals might be overwhelming at some point, with PIs still in proposal mode versus publication mode. I wonder if sometimes you hold back on publishing because it is easier to get killed on an experiment not yet done than on one mostly done. My feeling is that there will still be A0/A1 then new. This is reasonable- you should be able to build on review. Obviously no guarantee that you can get anything out of response to review, but in some cases those conversations make the grants sharper and the previous reviews and the response allow things to be put in contrast that are less pronounced on a clean slate. I think that this doesn’t have too much of an impact, but will help people in the 20-50 range feel like they can keep working and not arbitrarily have to trash likely or possibly reasonable stuff.
I need to think about it some more.
LikeLike
April 8, 2014 at 6:48 pm
Carlo M Croce has more than dodgy grant funding mechanisms in his arsenal. Just ask Clare Francis and/or the comment threads at Retraction Watch blog, if you get my drift.
LikeLike
April 9, 2014 at 4:25 am
I don’t necessary think this would be a bad thing, as it can’t be much worse that the two-strikes you’re out system we currently have. Well, a one-and-done system would be worse, hopefully THAT never happens. I had an A0 that was scored at 49, then the A1 improved to a 36. At that point I had to creatively re-jigger the grant based on one aim the reviewers liked to submit a new A0. That A0 got a 48. I reworked the approach and developed and published on an entire new method to address reviewer concerns, and the A1 got a whopping 37! My PO actually said I should be proud of that score! The reviewers suggested I take one of the new aims on a different subject than the original A0 to submit a new A0, so another round of grant yo-yo is in the offing.
The problem with two-and-done is that you basically only get one chance to have useful peer-review from the study section, and if your A1 doesn’t anticipate every single problem that could occur in a compelling 12-page masterpiece or you get new A1 reviewers that have a different frame of reference as your A0 reviewers, you’re sunk. At least without the substantial revision jazz there is at least a snowball’s chance that you can get an important project eventually funded. Honestly, I would rather be cued and get funded on A2 or A3 than have to revamp the entire project and continue yo-yo’ing and never get funded.
One thing I noticed from a bit of NIH Reporter sleuthing is that the study section I submit to funds full professors at over twice the rate of assistant and associate professors combined. I had to dig up the rank information since it’s not in Reporter, and my PO didn’t believe me when I mentioned the rank distribution. Either the full professors are just that much smarter than the assistant or associate professors, or there’s something else going on. Probably being well known in the field mitigates a lot of reviewer concerns about their ability to pull a project off that sink people who had the bad judgment to be born later.
LikeLike
April 9, 2014 at 4:53 am
I’ve said before that all this effort the NIH puts into enforcing the A1 rule is a waste of time since everyone seems to get around this anyway. What appears to be an “A1” or even an “A0” could be the result of 4-5 prior submissions under different mechanisms or a different disguise. Removing the useless submission limit will have no impact on success rate, and may uncover what the time-to-funding really is.
LikeLike
April 9, 2014 at 4:53 am
@Pinko: “I wonder if it will just lead to an expectation of massive amounts of preliminary data, because some proposals might be overwhelming at some point, with PIs still in proposal mode versus publication mode. I wonder if sometimes you hold back on publishing because it is easier to get killed on an experiment not yet done than on one mostly done.”
Yeah. So, in other words, it becomes necessary to do the research first in order to get the grant, and unlimited resubmissions just makes that true in more cases. Eventually, maybe the NIH will actually formalize it, and do what I’ve been suggesting all along: award grants based on past results, not applicants’ bullshit about the future.
@Viroprof: “the study section I submit to funds full professors at over twice the rate of assistant and associate professors combined”
I fund that completely unsurprising. Full profs have learned how to write convincing grants, they have the resources to gather enough preliminary data (see above – what that means is, they can do the experiments first), and the name recognition (and often pals on the study section) to ease fears about whether they can do what they said they are going to do.
So, again, why not just formalize this? Give people who have established records some amount of funding based on their records, so they don’t have to waste time with grants.
LikeLike
April 9, 2014 at 5:20 am
I don’t know how many people actually got stuck in the filter for submitting a A0 that was too similar to their prior, unfunded A1. I heard of a few, so it did happen. On the flip side of that, I’ve sure as heck been putting in more than two versions of a proposal which is designed to fund the same area of interest in my laboratory. I have not yet been flagged for it. My initial reaction that any PI who has an ounce of creativity ought to be able to come up with a credible alternative take on their project is still my current take.
More dangerously, the will of study sections to endlessly queue applications will be released from whatever tepid effect the A1 limit has produced.
This is a very BadThing.
Dude, you need to get your story straight. How is repeal of something that has only had a “tepid effect” a “very BadThing”?
LikeLike
April 9, 2014 at 6:03 am
@Grumble – Are you suggesting that once someone has reached greybeard/hair status that they just get a dedicated funding line from their IC and let the kids fight it out for the remaining scraps? I would guess that going this route would result in a lot of unemployed junior faculty within a few years as more associates reach that status and senior faculty keep plugging along, making the available pot smaller each year and increasing the viciousness of review. Come to think of it, this does seem to be how it currently works.
I wasn’t terribly surprised at the funding distribution either. I agree that senior faculty may have learned to sell their work better, but I have read funded grants from senior faculty that would have been quickly triaged if I had submitted it. It seems to be an unwritten rule that you get extra points for being a BSD. I need to figure out how to become a BSD. I hear there are pills for that.
LikeLike
April 9, 2014 at 7:02 am
Given the fact that it seems all my grants got 4 reviewers and that I have read crap like “He answered well to all the prior reviewer’s concerns but I have now these concerns…” (knowing very well that I had no chance anymore to respond to his concerns) having a chance to turn it around quickly and resubmit to another round and/or study section for example is a good thing. Currently the reviews are so much at BS noise level (a colleague of mine got the comment in a A1 that he should come back when he has actually done the research he was applying for…) that it increases the chances of getting two reasonable reviewers if I can go straight back.
LikeLike
April 9, 2014 at 7:59 am
PP- your powers of logic are leaving you. Probably all that hanging around with “free thinkers” is dulling your mind.
LikeLike
April 9, 2014 at 5:43 pm
CPP, DM, CPP.
LikeLike
April 10, 2014 at 4:01 am
No, this is a GoodThing.
First off, even you DM admit to shenanigans to circumvent the ‘substantially different’ rule. So if you’re not going to play according to the spirit of the rules, why defend them?
Second, the two strikes you’re out policy does definitely penalize junior investigators, who still don’t have the track record and experience to nail a proposal first time around. It also penalizes investigators at smaller schools, who don’t have the infrastructure and colleagues to enable bleeding-edge preliminary data in an A0. The two-strikes rule just helps the rich get richer.
Third, If NIH is all about funding the best science, then what’s wrong with multiple rounds of input from experts in your field? How does lots of evaluation and revision make the eventually funded science worse?
Fourth, it might actually shift applicant emphasis to perfecting a particular project rather than throwing shit at the wall to see what sticks. I wouldn’t mind a rule whereby a PI could only have a very limited number of proposals in the system at any one time. The NSF BIO directorate sets the limit at two.
LikeLike
April 10, 2014 at 4:50 am
I like the sound of this. “Everything an A0” means less nitpicking over precisely when and at what stage to start submitting.
I’ve got a new and very promising project going with about 70% of the proof-we-can-do-what-we-say-we’ll-do preliminary data. We won’t have the remaining 30% by June, which means skipping a round if we don’t want to waste one of our two chances.
In an all A0 regime, there would be no reason not to go ahead and submit and get early feedback about what
irrational bees in their bonnetsinvaluable scientific advice we can expect from the study section.LikeLike
April 10, 2014 at 5:51 am
@Viro: “Are you suggesting that once someone has reached greybeard/hair status that they just get a dedicated funding line from their IC and let the kids fight it out for the remaining scraps?”
Not at all. I’m suggesting that established people with good to excellent track records be given a certain minimum amount of funding without the need to describe in detail what they are going to do with it. The amount could be capped so that it is less than what the grayhairs get now. So, the amount of money leftover for younger folk via competitive mechanisms could actually be more than it is now, not “scraps”.
One way to do this would be to offer productive, established PIs a choice: $X via a mechanism that doesn’t require an extensive detailed proposal, or a chance at some amount >$X via a more traditional competitive mechanism (and if they fail to get it, they don’t get the “free” $X). If $X were, say 10 to 20% less than the average amount a grayhair PI typically gets, I suspect that many PI’s would just take $X. That would increase the money “left over” for other mechanisms, and help to even out the spread of money between the young bloods and the old farts.
Finally, although I referred to $X as “free”, it wouldn’t be, really: it would be based on evaluation of the PI’s overall track record. Low past productivity=low or no $X.
LikeLike
April 10, 2014 at 8:40 am
@Grumble: Your plan is already in place. People with good-to-excellent track records get all sorts of forgiveness with regard to their crappy proposals. And thus, they basically have guaranteed income. Others struggle.
LikeLike
April 10, 2014 at 12:12 pm
I think Grumble simply does not like the current dollar value of the “free” $X money for established PIs, at least as a percentage of effort in grant submission. Its just that the money:grant effort is even worse for people starting out.
Really the thought of anyone getting “free” money makes me queasy. The problem is not that competition is bad, the problem is over-limited resources.
LikeLike
April 10, 2014 at 12:31 pm
I doubt it will change things very much. NSF (at least the panel I reviewed for) allows infinite resubmissions and it’s not terrible. (Whether it’s worse than NIH I dont know). I suspect it winds up not mattering much.
The sad part, as a reviewer, is the number of people who submit revisions without really responding to the comments. It seems as if their institution rewards them for submitting and they dont actually care if they get it.
LikeLike
April 10, 2014 at 2:14 pm
It seems as if their institution rewards them for submitting and they dont actually care if they get it
I hope you are kidding. this is unimaginable.
LikeLike
April 11, 2014 at 6:08 am
I recall Jeremy Berg saying that he had seen NIH data presented in re the degree to which resubmitted grants fair better than their original submissions. I think that data needs to be publicized before there can be a coherent conversation about whether the opportunity to respond to critiques is worth the while, or whether the simpler NSF model might be adequate. Intuitively, one would predict that, on aggregate, resubmissions should score better than original submissions, but there does appear to be a preponderance of anecdata lately that would suggest otherwise.
“The sad part, as a reviewer, is the number of people who submit revisions without really responding to the comments. It seems as if their institution rewards them for submitting and they dont actually care if they get it.”
I think this is why there needs to be an evaluation of whether responses to critiques are really enhancing an applicant’s chances. I’m willing to bet that this seemingly cavalier behavior is chiefly committed by veterans of NSF funding, who have likely figured out through experience that beyond a certain basic point of grantsmanship quality, reviewer criticisms become rather fickle and arbitrary. In which case, spending considerable time responding to a reviewer’s particular subjective eccentricities and criticisms is no longer the smart option given that you’re guaranteed a completely different reviewer with a different set of subjective eccentricities the next time around. Under such circumstances, it’s not surprising that some PIs just submit the same lottery ticket and be done with it.
Of course, this might by flawed and cynical thinking. There might indeed be clear and compelling evidence that responding to criticisms really enhances the fundability of an application. If this is the case, it should be easy enough to produce a graph demonstrating that fact based on the last decade’s worth of scores, whether for the NIH or the NSF.
LikeLike
April 11, 2014 at 6:34 am
I hope you are kidding. this is unimaginable.
Actually, I can see how something like this might develop. The Chair and/or admins, realizing how tough funding is right now, decide that they’d end up gutting the entire department if the dumped everyone who wasn’t currently funded. Since they don’t want dead weight hanging around, they say “It’s OK (for now) if you’ve temporarily lost funding, but you need to be actively putting proposals”. “Grants submitted” then becomes part of what you’re evaluated on in your annual performance review, and viola!. Now you have people churning out crap proposals just to show that they’re “actively” seeking funding.
The solution, of course, is to require that people are putting proposals and getting competitive scores a significant fraction of the time.
LikeLike
April 11, 2014 at 7:28 am
sorry, yes, i am sure they do care that you get the funding.
but there are all sorts of situations where just simply applying buys you something. it proves you are trying at least! it proves you can get something out the door! and in small ways, that makes one’s life easier. it may not be enough to get tenure, but it might make a 3rd year review go smoother, it might make someone on the edge look a little bit better, it might be a (de facto if not de jure) expectation that one needs to do that if one gets an intramural award.
LikeLike
April 11, 2014 at 9:11 am
“The sad part, as a reviewer, is the number of people who submit revisions without really responding to the comments. It seems as if their institution rewards them for submitting and they dont actually care if they get it.”
I’d have to agree with dsks on this one. I have veteran NSF-funded colleagues who say they never respond to comments, because it doesn’t matter. My personal experience was having a proposal on the borderline for funding (Excellent, Excellent/very good, very good). I revised with respect to comments and the reviews got a little worse (though retaining one Excellent). The Program Director just said that different reviewers have different perspectives.
On the A2 or no A2 question, I have had an experience with CSR essentially killing two projects with one ‘Virtual A1’ decision. Project 1 went in as A0, scored OK, not great. In rethinking and getting more data, the questions, experiments and aims all changed dramatically (entirely new Aims), but the general subject area was the same, as were about half of the methods. So I sent it as a new project A0, planning to get some feedback on the project, and hoping to have a second shot with the A1. CSR deemed it a ‘Virtual A1.’ So it was reviewed as an A1, not funded, and is now dead. The consolation is that the Summary statement clearly indicated that the reviewers saw it as a new project. Somehow I’m less satisfied with a moral victory here.
The return of an A2 would help with this kind of thing, at least you’d have a better shot when CSR doesn’t get it right.
LikeLike
April 11, 2014 at 9:53 am
@TOD: “Your plan is already in place.”
Then why not formalize it and reduce the bullshit effort that established people have to put into it?
@anonymous postdoc: “I think Grumble simply does not like the current dollar value of the “free” $X money for established PIs, at least as a percentage of effort in grant submission. Its just that the money:grant effort is even worse for people starting out.”
I was “just starting out” not too long ago, and I have not seen the money:grant effort ratio increase by one bit. Also, I’ve seen lots of established people lose a portion of their funding. So, while it’s definitely no easier for the newbies, it’s getting harder for established people, too. Most importantly, I don’t think my proposal would actually make it worse for the young guns. As I said, it might make it better by reducing the old farts’ financial drain on the system (i.e., if a significant number of them accepts less money in exchange for more stability).
“Really the thought of anyone getting “free” money makes me queasy. The problem is not that competition is bad, the problem is over-limited resources.”
Which is why I put “free” in scare quotes. It wouldn’t be “free”, it would be based on recent past performance – and that would be judged in competition with an applicant’s peers. You’re right that competition isn’t bad, and I think it should still play a big role in how grant money is divided. But endless grantwriting for very little payoff is also bad. What I’m suggesting is a measure that would restore a degree of balance.
LikeLike
April 11, 2014 at 11:12 am
“The sad part, as a reviewer, is the number of people who submit revisions without really responding to the comments.”
Don’t know about NSF, but responding to comments is an important part of the resubmission for NIH. There is a good chance the A1 is going to the same reviewers or some of the same reviewers, and they are going to be ticked-off if you didn’t address their concerns. Also, it’s a free page for you to address concerns as well as get in some more info on how awesome your project it.
LikeLike
April 11, 2014 at 12:11 pm
I’ve personally been hit by ‘this A0 has been deemed to be an A1’ twice, despite having different SAs, experiments, and framing. So that’s certainly been in play. [I was able to pull from being reviewed as an A1-without-response, though; went in the next cycle (actually the current one) as an A1 but at least with a response. We’ll see.
And: yes, my current institution formally rewards/acknowledges submitting many proposals even if none are funded…
LikeLike
April 11, 2014 at 1:02 pm
Having tried to get difficult projects funded at both NSF and NIH, I have found a real difference – having everything be new means that everything is a crap shoot, and there is no real reason to respond to criticisms unless they’ve really nailed you on something. With the A1 system, you have an idea what the reviewers are looking for. Every reviewer is a special snowflake, and you have to find the right key to fit that lock. With the “every grant is new” system, you never know what that lock looks like.
With the A1 system, responding to comments is the key to funding. It means that there’s a little air traffic control, but which is worse: a one year wait but a very good chance of getting funded or having a random X% chance (X being very very low) every time?
The thing that we keep sidestepping is that the “every grant is new” system moves us even more towards the “I have to submit 10 grants/year to have a reasonable chance to keep my lab funded.” The problem is that if there is a 10% chance of getting funded and 10 people submit 10 grants, then each person doesn’t get 1 grant. Some get 0, some get 2, and some get 5. Suddenly getting 5 grants at once is NOT a good thing. It’s not like winning the lottery. It’s like suddenly having 5x as much work to do, when you’ve already k3rned your 24/7 work schedule to the maximum possible.
And let’s not forget how this also wastes reviewer time, reviewing those extra 90 grants. Having a system which reduces the number of grants you have to submit (by providing you a revision system) reduces your variability (a very good thing), improves your ability to plan (a very good thing), decreases the amount of work for reviewers (a good thing), all at the expense of some applicants having to go an extra round (slightly bad). But what’s the balance?
PS. I’d take Grumble’s deal in a heatbeat!
LikeLike
April 11, 2014 at 1:50 pm
qaz,
NSF is plagued by it also not being clear which reviews are tossed in shredder and which are the most meaningful. Sometimes this can be made clear from the summary and sometimes not. NSF grants can have a lot of reviews if everyone asked to do ad hoc review accepts. I have heard of 6 or even 8 reviews. Given that there are 8 reviews from people that might only be looking at one grant, and are not on the panel and in discussion mode, there is a huge, huge variety of opinions. This is incredibly difficult to navigate. The value of discussion for grants that are not funded is that good or bad reviews can emerge or be mitigated, respectively by the panel in discussion. This is also why discussion of more than 40% of grants would be useful. Panels think it is not worth it, but it can buffer extreme outlier reviews that otherwise might be considered to have equal weight with no discussion. That said, if reviewers don’t edit their reviews to reflect their actual votes after discussion, they can be doing the submitter a huge disservice by not providing a correction to the record. Frustrating.
LikeLike
April 14, 2014 at 1:03 pm
They’re all new again after 37 months anyway, which is just enough time for your R01 A0 and A1, submission to an RFA, conversion to an R21 for that A0 and A1, and then back to NIH as a legit shiny “new” A0 R01.
LikeLike
April 17, 2014 at 5:35 am
Dude–I just got an email with this link:
http://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-074.html
LikeLike
April 17, 2014 at 6:44 am
Yep, the game is ON people.
LikeLike
April 17, 2014 at 7:56 am
[…] rumors were true. […]
LikeLike