Your Grant in Review: Incomplete Information

October 15, 2007

As we are in the middle of study section meetings for NIH grants submitted for the June-July dates and heading toward yet another revised-application due date, I’m thinking about the way amended applications are reviewed. The amount of information available to a given reviewer on the previous history of a particular amended application is variable, leading to much dissatisfaction on the part of the applicants. The system could stand to be improved.

When an amended NIH application (A1, A2) is reviewed, it picks up a 6th major category of critique, namely the degree to which it is “Responsive to Prior Critique”. There are some very important problems with this process, namely the delay for revision churning and the lack of effect it has on the eventual conduct of the science; I’ve touched on that before. In this post, however, I want to consider the more prosaic and practical implications of the way amended or revised applications are reviewed. The reviewer of the amended application is provided with the most recent summary statement and the revised application. Period. There is no provision of the actual prior application(s) or prior summary statements (for an A2 revision) and certainly not the prior score ranges, the identity of the prior reviewers or anything else.

The first and biggest problem is the failure to provide the prior application to the reviewer to facilitate assessment of “responsiveness” to the prior critique. This means that a new reviewer for the application is basically guessing on the basis of the summary statement what was contained in the prior application. [Grantsmanship sidebar: This is why it is a GoodIdea to follow the convention of highlighting changes in the application with a vertical black line in the margin next to substantially revised text passages.] S/he is also hampered in ability to determine what parts of the summary statement may be unjustified and therefore a failure to respond to a particular critique may be neutral. You can imagine first how difficult this is as a reviewer new to the application and second the degree to which summary statement writing which is too brief or too poorly matched to the actual review/discussion can introduce big problems.

The second problem is related and stems from the fact that revised applications not infrequently have reviewers who have reviewed prior versions, reviewers who may have been present at the discussion (but not have been assigned the application) and reviewers for whom the proposal is novel. Thus there will be a varying level of recollection of the prior version(s) of the application and of the issues that arose at the prior review. This can be really frustrating. For example when a set of critiques come back with some saying “highly responsive” and another saying “didn’t appropriately respond to the comments on X, Y and Z”. You can imagine the reasons. A previous reviewer may recall the key points of discussion better and be able to focus on the criticisms that are most important to be changed. Alternately a novel reviewer may not have any ego bound up in the process and may think an partial response perfectly appropriate whereas the person who made the comment is unsatisfied by anything less than wholesale adoption of the critique. My point is not to say that either scenario is always correct or always incorrect, just that this is a source of variance in the process.

Finally, a further related point is that even for reviewers who have reviewed the application before, their memory of what happened 8 months (the usual 2 review rounds necessary to revise) ago can vary tremendously. The official rules insist that reviewers are to destroy all information related to the review including copies of the applications and any notes made during review. I wonder how seriously people take this. I have for certain sure received summary statements of my revised applications in which substantial blocks of text are reproduced verbatim or nearly so. It takes no great leap to see that the return reviewers have held on to their prior critique document file for re-use on the revised application. Since they cannot reasonably judge which revisions they will later receive to review, it is likewise no leap to assume that reviewers may hold onto all of their reviews that they have written. Do they hold onto notes and the applications as well? Surely some of them do!

So there is clearly going to be a wide range in the familiarity a set of reviewers has with the history of, say, an A2 amended application. From a very intimate appreciation (reviewer has seen all three versions, has great memory for the review discussions and has retained the prior applications and notes) to essentially minimal historical information (a reviewer new to the panel and application). In and of itself this is bound to introduce variance into the process. As a related concern the difference in familiarity is going to bias the perceived authority of the reviewers in the impact it has on the rest of the panel. On my panel at least people are plenty forthright about saying “I’ve reviewed this one the prior two times and….” or “Well, I didn’t see the prior version but it seems responsive to me…”. This is going to have disproportional impact on the rest of the panel. As above, I’m not saying this is monolithic in direction. One might assume the clearly more-knowledgeable reviewer will seem more authoritative. However, sometimes the “more informed” reviewer comes across as the dog who will just not relinquish his/her favorite bone which the rest of the panel can see should long since have been buried.

There seems a very simple solution that would go a long ways towards reducing these sources of variance. Namely that all reviewers should be provided with all the prior versions and summary statements. This would go a long ways towards equalizing the amount of “extra” information a given reviewer brings to a revised application.

8 Responses to “Your Grant in Review: Incomplete Information”

  1. StruggleBunny Says:

    As a New Investigator, this material is very interesting.

    Here’s my situation: due to pressure from my chair to just get into the system, I submitted an 01 application while it was mostly ideas and not a lot of preliminary data. It scored badly (50+%) but it did get scored, with comments along the lines of: this is interesting, but you have to prove you can do it. For the A1 application, I got the experiments to work, had a pile of preliminary data, and got a high impact factor (10+) publication based on this new work. The A1 was triaged. Now what? I’m thinking it’s not going to be possible to go from a triaged A1 to a funded A2. The message I’m getting is that the study section is just not interested, or, at least the primary and secondary reviewers aren’t interested. The study section it went to however is very highly biochemical and mechanistic, and these experiments are more functional genetics than biochemical mechanisms. Based on roster lists, some input from other people, and a CRISP search of grants that individual study sections have funded in the recent past, I think I’ve identified a study section more likely to be receptive, but I still need to talk to the Program Officer about this.

    So, do I send the A2 to a study section that has never seen this application before, or do I rework the whole thing and have a do-over as a fresh 01 application? For new investigators, revised applications can be rapidly resubmitted without missing a review cycle, whereas a new application would need to go in the review cycle after that, which is a consideration with startup funds running low.

    If an A2 goes to a new study section, should I address explicitly in the introduction why I’ve sent it somewhere new? I think sending it back to the original study section is going to be a waste of everyone’s time and effort. Should I break the R01 into pieces and submit a bunch of R03 and R21 applications instead? Is is allowable to both submit an R01 and derivative R03/R21 applications simultaneously?

    Frustratingly, it’s been a week since it was unscored, and still no critiques. Since the application was unscored, there’s no discussion to summarize, so why doesn’t the SRA just release the comments directly? Didn’t the SRA get these comments in advance? The delay is irksome and for a rapid turnaround grant every day is precious.

    Like

  2. physioprof Says:

    “I’m thinking it’s not going to be possible to go from a triaged A1 to a funded A2.”

    Not necessarily. Just like your application unpredictably got scored worse as an A1, even though you clearly improved it, it could get scored dramatically better as an A2, even if you don’t improve it that much.

    “So, do I send the A2 to a study section that has never seen this application before, or do I rework the whole thing and have a do-over as a fresh 01 application?”

    Neither. You might as well take one more shot with the current study section and take advantage of the “A2 bump”.

    “If an A2 goes to a new study section, should I address explicitly in the introduction why I’ve sent it somewhere new?”

    I wouldn’t try to get it sent to a new study section, as there it is not likely you will get the “A2 bump”, because the new study section will not consider itself to have an “investment” in the application.

    “Should I break the R01 into pieces and submit a bunch of R03 and R21 applications instead?”

    Absolutely not. Except under very special circumstances, R21 and R03 applications are a waste of time and effort. They take almost as much time and effort to write as an R01, are reviewed just as mercilessly, and in the end give you hardly any money and, thus, career advancement credit. And they can’t be renewed competitively. In order to advance your career, you need R01s, period.

    “Is is allowable to both submit an R01 and derivative R03/R21 applications simultaneously?”

    I’m not sure about this. Guaranteed Writedit knows.

    “Since the application was unscored, there’s no discussion to summarize, so why doesn’t the SRA just release the comments directly? Didn’t the SRA get these comments in advance?”

    The SRO (they’re now called Scientific Review Officers) needs to do at least a cursory review of the reviews to make sure that there is no inappropriate language or personally identifying information. From what I have been told, they are supposed to start with new investigators, and then go in priority score order. I’ll bet you’ll get the Summary Statement soon.

    Like

  3. drugmonkey Says:

    “Except under very special circumstances, R21 and R03 applications are a waste of time and effort. They take almost as much time and effort to write as an R01, are reviewed just as mercilessly, and in the end give you hardly any money”

    I definitely agree with the sentiments. I would refine it a bit to suggest that if the R03 or R21 is being chosen as an alternative to the R01 then no, don’t do this. If, however, you have a full R01 app or two in process then it can be a good idea to spread out in terms of mechanism. Just make sure the ideas and plan really fit.

    First, money is money after all. And if you happen to have a FTE position and cheapish research, even $50K for two years can be decent. Enough to get a pub, for instance and certainly enough to generate some really good prelim data. I’ve learned to work with R21 size efforts despite the fact that I’m soft money (so R21 cannot be a replacement for R01 in any real sense) and I have colleagues that at times have held something like 3 R21s simultaneously (which approximates small R01 money). So it can work and is not necessarily “cutting yourself off at the knees” depending on circumstance.

    While for the most part PhysioProf has it right that panels have a pronounced tendency to review everything like an R01, there are exceptions. So you may get lucky and hit on that trio of reviewers that really gets what “Exploratory/Developmental” is supposed to mean. Or that hard acre SRO that pulls the panel into line. It can mean the difference.

    You should have, in your mind, at any given point in time a series of R01, R21 and R03 type thoughts for research plans. If it is really an R21 type project, why try to tart it up as an R01? Reviewers will see through this. So that particular idea either has to wait for full development, prelim, etc or you can write an R21.

    A consideration I’ve touched on before (see above link) is the fact that we still have people who think that new/young PIs should have “starter” grants. Should start with a dinky award to “prove” that they won’t “waste” it. Gak. Nevertheless, it is a reality. A PI I know just got some PO scuttlebutt from the review that emphasized this. It boiled down to “well geez, the applicant just got a first R01 funded, tell ’em to settle down stop putting out so many apps and work on that project for awhile”. While this sort of crap makes the steam come out of my ears, it is a Fact o Life. People who think like this are more likely to look favorably on a smaller “second” grant than another big grant.

    with respect to “derivative” submission, well the bottom line is that you can not have apps that are “substantially similar” (or some such verbage) under consideration at the same time. this should be no obstacle to anyone as far as I can see. The point is to simply take a sufficiently different tack that it is not obviously overlapping. So no, you can’t just transplant Aim 2! I just don’t see why it would be difficult to gin up a reasonably different approach to a similar project to fit R03, R21 and/or R01 type apps.

    And keep in mind that this is a skill you may need to acquire for the “A3” submission. “Hey wait, they don’t allow A3s”, you say. Well yes but what are you going to do when your A2 goes down in flames? Give up? Maybe. Or maybe you will retool it a bit and create a “new” application and take three more shots!

    Like

  4. gnupi Says:

    As a new PI one of my frustrations with the NIH grant system is the appearance of randonmess or even chaos in the process. Reviewer turnover and the inability of reviewers to see important information about a resubmitted grant just adds to this problem.

    I propose a hypothetical grant experiment. In science we like to repeat things at least 3 times, right? If you submitted the exact same grant simultaneously (of course you cannot, but let’s imagine) say to 3 study sections essentially equally capable of reviewing it, how close would the scores be? What variables would determine how close the scores were to each other.

    Variables (in no particular order)

    –“friends” on the study section
    –seniority of the applicant
    –reviewer turnover
    –current other funding (perceived need?)
    –A0, A1, A2 (which would produce more variable score?)
    –quality of the grant (does this even matter? No I’m not be facetious) Would a poor grant or outstanding grant result in more variable scores

    What do you think? How would you arrange these variables and can you add others? Are any of these in our control?

    Like

  5. BugDoc Says:

    In deciding about whether to apply for an R21 vs. R01, I thought it used to be that very little to no preliminary data was expected for an R21 app, hence the “exploratory” label. However, in a recent round of review, the major criticism for a colleague’s R21 app was that reviewers wanted to see much more preliminary data. So practically speaking, there may be less difference between the R21 and the R01 than there used to be. I think this sort of reviewer comment brings up an interesting point. Where is the funding to generate all of these preliminary data supposed to come from? R21s? I guess not. With regards to that question, I also thought it was ironic that a recent study discussed in the Oct. 3rd issue of Nature shows (surprisingly!) that researchers are rather “lax” when it comes to accounting for % effort. I wonder how much this is due to the fact that we have to “rob Peter to pay Paul” and use currently available research funds to develop preliminary data for as yet unfunded grants.

    On a related note, here at R1U, the university pays 50% or less of our salary and we pay all research costs, including about $50K worth of stipend and tuition per student per year, even though we are the ones teaching and mentoring them. So the university gets the prestige of an institution of advanced education and research, for pennies on the dollar. Hmmm, maybe I won’t apply for an R21 after all.

    Like

  6. drugmonkey Says:

    “So practically speaking, there may be less difference between the R21 and the R01 than there used to be.”

    I don’t know that it has ever been any different but the inability of study sections to genuinely review R21s differently than they review R01s is a common criticism. Program is aware of this and partially offsets the problem by picking up R21s at scores that wouldn’t float an R01, of course.

    “we pay all research costs, including about $50K worth of stipend and tuition per student per year, even though we are the ones teaching and mentoring them. So the university gets the prestige of an institution of advanced education and research, for pennies on the dollar.”

    err, you do recall that the grant is awarded to your institution on behalf of you as PI, right? Technically, your grant is your university’s money.

    I bet you wonder where your overhead $$ go too, don’t you? 🙂

    Like

  7. bikemonkey Says:

    “As a new PI one of my frustrations with the NIH grant system is the appearance of randonmess or even chaos in the process.”

    It is not just new PIs who are frustrated with this random-mess. Some of us have come to the very considered opinion that the only logical response is the shotgun method in which one tries to get one’s applications in front of as many different reviewers and panels as possible. To revise and resubmit each one down to the A2. To create approaches of interest to as many of the ICs as one can. Etc.

    This doesn’t mean shoot out a bunch of poorly written garbage apps, of course. You don’t EVER want to give a reviewer the idea that you are not taking his/her time seriously.

    Like

  8. physioprof Says:

    “Some of us have come to the very considered opinion that the only logical response is the shotgun method in which one tries to get one’s applications in front of as many different reviewers and panels as possible. To revise and resubmit each one down to the A2. To create approaches of interest to as many of the ICs as one can. Etc.”

    BINGO!!

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: