Ask DrugMonkey: Has the revision strategy changed for NIH Grants in the A1 era?

May 27, 2010

Occasional commenter crystaldoc left a query on an early post.

Given recent changes in the NIH process (in particular only one chance for resubmission, and less information in the tea leaves of the new summary statements), when does it make sense to resubmit? When does it make more sense to change it up and put in a new submission? Any pragmatic advice or guidelines based on impact scores or percentiles? How often are A1’s funded when the original submission was streamlined? Or at 40-50th percentile, or 30-40th percentile? Are these data available anywhere?

I offered up a prior post in which I posted some longitudinal data on the funding of NIH grants unrevised and at the A1 and A2 revision stages.
crystaldoc was not impressed.


We can perhaps get a little closer by following this link [ppt] that I included in that prior post. It shows the fate of unsolicited R01 grant applications submitted in 1998, 2004, 2005 and 2006 by the percentile rank of the original submission.

UnsolicR01fate-2006.png

Now unfortunately data such as these are hard to find in the plethora of NIH powerpoint slides scattered around here and there. In addition, these tend to come out with a time lag. So we always have to recognize that when circumstances are changing (such as the new limit to a single revision and assorted efforts to revamp the grant review) our ability to predict the future is poor. And of course when we are viewing NIH-wide stats we need to recall that specific study sections or even funding institutes may represent outliers from the general trends.
Still, I always contend that longitudinal trends can help us to identify some basic advice.
I consider this from the data slide. Out of the original submissions in 2006 which garnered a 50%ile rank some 11% or so were funded as A1 revisions and another ~12% were funded as A2 revisions.
So there was a nonzero chance of the revisions getting funded. It isn’t in this slide but we know from other data (and common sense) that some fraction of those original submissions at the 50%ile did not come back in as revised A1 grants and some further fraction of those did not come back as A2 revisions.
I think you still have to use your available revision cycles. It is crazy not to because that is the only way to guarantee it will not be funded.
Now, crystaldoc alluded to effort, whether it would be better to work on a new submission rather than revising. My point here is that there is a month between the new-submission receipt date and the revised-application receipt date. You really should be able to revise a grant in a month so I’m not seeing the conflict.
All things equal of course. You need to think for yourself how the balance of revising a grant, writing up another paper or generating data works out in your own specific circumstances. (Just don’t overcredit the value of another paper or more preliminary data, as is my constant refrain.)

About these ads

25 Responses to “Ask DrugMonkey: Has the revision strategy changed for NIH Grants in the A1 era?”

  1. zoubl Says:

    “You really should be able to revise a grant in a month so I’m not seeing the conflict.”
    yes, sure a month is enough. But the SRO never or rarely releases the summary statements until a week before the deadline. I don’t understand why we can see the score only a week after the study section meets, but it takes forever to post the comments.

    Like

  2. DrugMonkey Says:

    The NIH has been working on speeding up the return of summary statements in recent years, I will note. Particularly for the New Investigators and ESI applications, I can totally get behind prioritizing these folks.
    The process, at least in general, is that reviewers have an interval (used to be a week, they’ve tightened that up to a couple of working days) after the meeting to do some final revisions and editing of the review. Then the SRO has to compile each one, write a resume of discussion, make sure the review language is not egregious in a number of categories, etc. S/he has a lot of these to write up- 30? 50? something in there is probably the average for regular study sections. More for the current round, I understand the post-ARRA deluge was as notable as expected.

    Like

  3. James Says:

    I never understand why score distributions from study sections aren’t readily available. The scores are basically set by the end of the study section and don’t need editing–the numbers are the numbers. Can’t someone make a little app that the SRO can upload the score distributions within a week of the meeting and it would display a happy little histogram right on your Commons page next to your priority score? I realize applications from the SRO get split to different Institutes so the pay lines will be different, but I still think it is useful to see where land relative to the group you were reviewed with.

    Like

  4. Physician Scientist Says:

    I interpret the data in the following manner. The cutoff score for resubmit as an A1 is probably 40. With an initial score of a 40, the resubmission has about a 50% chance of getting funded. Given paylines at the 10-15%, this is a good use of time. Lower than 40, and I think the numbers probably support revising as a new grant application.

    Like

  5. JAT Says:

    Z and J
    SROs do not upload the scores. They must turn over to the assistants to upload. It is NIH rule, not SROs’ choice.
    SRO now gets one month (as supposed to three months in the old days) to deal with all the summaries (40-50% of them require resume). That is not a whole lot of time if there is more than 80 applications on ave each round for most study sections. As DM mentioned, NEI get their summaries at the end of one week after the meeting is done. That means they only have three weeks to finish the rest. I am an applicant and I want my summary statement fast as well. But I rather have a well written resume (especially now we don’t get much of anything from the reviewers) than one that comes out of the same mode. Believe me, it is a very difficult task the SROs are made to do with shortened turn around time. For me, unless my percentile is a just miss, there is no way I will ever want to resubmit to the next immediate cycle like the NEI. So it does not make much difference if you get your summary in two weeks or in four weeks (note: generally, if you are unscored, you will get yours much later than the ones who got scores because SRO must make sure the resumes are done first before going for those that need only polish in the language)

    Like

  6. zoubl Says:

    Thanks for the feedback. What accounts for the 4-month wait between submission of a new application and the study section meeting? If this timing was reduced, wouldn’t this ease the burden on the SROs and the assistants?

    Like

  7. crystaldoc Says:

    Thanks for highlighting this slide; I’d missed it before and it does offer some useful data points. Still, I can’t help but think odds are much worse for a 40-50%-ranked A0 on its A1 resubmission now compared to 2006. I mean, aren’t there way more good grants, backed up and circling in a holding pattern, because they haven’t been getting funded? It seems like the cumulative effect of sustained low paylines will get worse and worse over time. My colleague who is reviewing R01s right now is complaining what a waste of time it is for him to have to review resubmissions of grants that were triaged or got a score worse than 30% on initial submission, because he believes there is little to no chance they will be improved enough to be fundable. At least, that is his anecdotal experience in his particular study section.

    Like

  8. DrugMonkey Says:

    A good month, month and a half of that is for the reviewers to do the reviewing. On the front end it takes a week or two minimum for the wheels to churn in assigning apps properly to institutes for potential funding and to study sections for review. Remember we’re talking a lot of apps. I don’t know how much is automatic by keyword, how much is manual and how much is back checking automatic assignment but I can see where it might take two weeks to get this right.
    In between, the SRO has to assign grants, round up new ad hoc reviewers, balance loads and expertise, etc. Throw in a couple of phone-tags trying to get reviewers and…
    I dunno, I guess you might be able to tighten that up a bit but when you think about actually having to do the job, 4 mo doesn’t seem outlandish to me.

    Like

  9. James Says:

    JAT-
    OK, fine, why can’t the assistants upload the distributions?
    My point is that the numbers and score distributions are there immediately after the study section and don’t need polishing. The distributions are what they are and those *could* be available sooner.

    Like

  10. DrugMonkey Says:

    My colleague who is reviewing R01s right now is complaining what a waste of time it is for him to have to review resubmissions of grants that were triaged or got a score worse than 30% on initial submission, because he believes there is little to no chance they will be improved enough to be fundable.
    Well that sounds like a self-fulfilling prophesy, doesn’t it?
    I don’t think I’ve ever had a grant that was initially triaged get funded but of course some of that is bias in what I chose to pursue.
    I have at least one that went from 51%ile to funded and another that went from 48%ile to a very close miss on the funding line
    I have not tracked the outcomes in study sections I’ve been on to make any strong statements. My impression is that it is not at all unusual for a grant to get really hammered, even triaged, and come back as being in play.
    with respect to the holding pattern, the grandfathered apps that can come back as A2s are almost all cleared out by now. ARRA funds relieved a good deal of pressure as well. I think we’re in a new era and it remains to be seen just what happens with unfunded A1s.
    CSR seems to think they can quash the thinly-revised new grants. I have my doubts but if they get aggressive they can do it. Lot of person power to review the submissions for similarity, however. Changing titles and Aims is not really that difficult.

    Like

  11. DrugMonkey Says:

    I still think it is useful to see where land relative to the group you were reviewed with.
    Useful for what, exactly? Beyond geeky interest in the mechanics of review (a charge to which I cop), all I can see is a vast potential for PIs kvetching about how they were mistreated or about how they deserve a pickup or something. Program is not keen on increasing that particular joy of their job…

    Like


  12. (1) The last submission date for grandfathered A2s is this November.
    (2) You have ignored a crucial strategic consideration, which is that some ICs are using looser paylines for A0s than for A1s.

    Like

  13. JAT Says:

    J-
    The assistants are like you and me…they don’t feel your pain! ouch
    I am very interested in knowing how to turn an outstanding unfundable grant into a new grant. Changing titles and aims are not enough. You should check out the guidelines recently (not sure if really is recent) published by NIH about “definition of new grant”. It is damn scary. I have heard that CSR will do as aggressive as they can to “take out” the “cheaters”. I just hope they will get tired of doing it after a while..but who knows.

    Like

  14. JAT Says:

    J-
    I meant The assistants are NOT like you and me…they don’t feel your pain! oops!

    Like

  15. neurowoman Says:

    Count me from triaged to funded in one easy step… Yes, maybe minor revisions won’t get you there, but if you haul ass and make major overhaul, along with significantly more preliminary data, another pub, etc, it’s worthwhile to resubmit an application that’s been triaged. It’s understandable that those numbers in the graph are daunting, but it’s not the stats you should worry about, it’s your grant writing and the data. What’s the point in submitting a new, but equally unfundable, grant application? For what it’s worth, I took a full year to resubmit (ok, so some of that was giving birth and all, but still).

    Like

  16. mk Says:

    Could you explain what A0, A1 and A2 refer to? I’m new to this grantspeak and need a translator.

    Like

  17. whimple Says:

    Count me from triaged to funded in one easy step…
    This has very little to do with how much you improved your grant and everything to do with who rotated off the study section in the interval. Specific people rotating off the study section can easily account for a 30% swing or more.

    Like

  18. DrugMonkey Says:

    mk,
    the NIH permits you to revise and resubmit a grant application if it is not funded the first time. These are termed “amended” applications and the identification number is modified to indicate the number of sequential amended versions, hence A1, A2. Long ago you could keep amending and I’ve seen A5s funded. But in recent memory the revisions limited to two. At present new submissions are limited to the single revision. A0 isn’t actually used by NIH to identify applications but it is used by some as shorthand for the original version of a proposal.

    Like

  19. crystaldoc Says:

    What’s the point in submitting a new, but equally unfundable, grant application?

    Well, obviously one wouldn’t, if one believed the new grant to be equally unfundable. But my perception is that fundable/unfundable has a whole lot to do with a grant getting targeted to the most appropriate study section. Getting a grant funded may in some cases have less to do with rewriting and more to do with getting it in front of a different audience.
    Here is (I think) a not uncommon scenario, particularly for new investigators with limited experience with various study sections, still trying to find the best “home” for one’s research, where one’s ideas find traction with the mindset of the reviewers:
    An initial submission to study section A is to some extent testing the waters. Summary statement from study section A suggests that this grant is unlikely to be funded here. After further consideration and discussions with more experienced colleagues, applicant believes the grant may have a better shot at study section B. In the old days, one could try to submit an A1 with a change of study section; this would give a chance to get a round of feedback from study section B, which could then be addressed in an A2. Nowadays, it seems a little crazy to fight for a change of assignment of an A1, since the only chance for funding is that the new study section loves the grant exactly as it is. Alternatively, one writes a new grant incorporating elements of the original grant, and targets it to study section B as a new submission.

    Like

  20. DrugMonkey Says:

    crystaldoc, make sure not to conflate gettng triaged with study section appropriateness. I’ve accumulated the most triages from a section that also gave me several fundable scores. As it should be- the most appropriate section is going to see a lot of apps from a given PI

    Like

  21. crystaldoc Says:

    Of course, a grant can be triaged or poorly scored despite being sent to the most appropriate study section. The science can be flawed. The models can be poorly chosen or lacking in physiological relevance. The hypothesis can be inadequately supported. There can be insufficient preliminary data to convince reviewers of technical feasibility. There can be inadequate evidence of productivity in a prior granting period. Any host of other problems. Even just bad luck in drawing fucking reviewer 3.
    None of this changes my point, that sometimes a grant may be mis-targeted, and as a result score more poorly than it probably would have, had it gone to a different study section. I suspect that this happens more often to n00bs, who haven’t yet figured out the ins and outs of different study sections. Sure there are objectively bad grants that would and should be triaged at any study section. At the same time, it is not enough to write a technically competent grant; there are quite a lot of those and only a subset will be fundable. You need to get the reviewers excited about your science, looking for reasons to advocate for you, as opposed to looking for excuses to lower your score to improve the outlook for other grants that they did find more exciting. To get these more subjective factors working in your favor, it sure helps to be at the right study section.

    Like

  22. DrugMonkey Says:

    You started this discussion by asking about revising an application versus writing a new one, unmodified by considerations of study section fit. You have now morphed into this discussion of fit.
    I’m just trying to keep it nice and clear in everyone’s mind that these are distinct issues.
    Sometimes you have the right fit and you get triaged. Should you revise? I say yes.

    Like


  23. I am aware of multiple instances of initially triaged grants getting funded on the next submission.

    Like

  24. Physician Scientist Says:

    I think the triage to funded is much easier if you are an established investigator and if the grant is a competing renewal. I know of a 10 year R01 that had a 52.5 percentile on an A1 that went to a 2.5 on an A2. Clearly, there is a queue in some study sections, but the key is to figure out if you’re queued or if you’re not in the running and submit appropriately.

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 40 other followers

%d bloggers like this: