Your Grant in Review: Incomplete Information

As we are in the middle of study section meetings for NIH grants submitted for the June-July dates and heading toward yet another revised-application due date, I’m thinking about the way amended applications are reviewed. The amount of information available to a given reviewer on the previous history of a particular amended application is variable, leading to much dissatisfaction on the part of the applicants. The system could stand to be improved.

When an amended NIH application (A1, A2) is reviewed, it picks up a 6th major category of critique, namely the degree to which it is “Responsive to Prior Critique”. There are some very important problems with this process, namely the delay for revision churning and the lack of effect it has on the eventual conduct of the science; I’ve touched on that before. In this post, however, I want to consider the more prosaic and practical implications of the way amended or revised applications are reviewed. The reviewer of the amended application is provided with the most recent summary statement and the revised application. Period. There is no provision of the actual prior application(s) or prior summary statements (for an A2 revision) and certainly not the prior score ranges, the identity of the prior reviewers or anything else.

The first and biggest problem is the failure to provide the prior application to the reviewer to facilitate assessment of “responsiveness” to the prior critique. This means that a new reviewer for the application is basically guessing on the basis of the summary statement what was contained in the prior application. [Grantsmanship sidebar: This is why it is a GoodIdea to follow the convention of highlighting changes in the application with a vertical black line in the margin next to substantially revised text passages.] S/he is also hampered in ability to determine what parts of the summary statement may be unjustified and therefore a failure to respond to a particular critique may be neutral. You can imagine first how difficult this is as a reviewer new to the application and second the degree to which summary statement writing which is too brief or too poorly matched to the actual review/discussion can introduce big problems.

The second problem is related and stems from the fact that revised applications not infrequently have reviewers who have reviewed prior versions, reviewers who may have been present at the discussion (but not have been assigned the application) and reviewers for whom the proposal is novel. Thus there will be a varying level of recollection of the prior version(s) of the application and of the issues that arose at the prior review. This can be really frustrating. For example when a set of critiques come back with some saying “highly responsive” and another saying “didn’t appropriately respond to the comments on X, Y and Z”. You can imagine the reasons. A previous reviewer may recall the key points of discussion better and be able to focus on the criticisms that are most important to be changed. Alternately a novel reviewer may not have any ego bound up in the process and may think an partial response perfectly appropriate whereas the person who made the comment is unsatisfied by anything less than wholesale adoption of the critique. My point is not to say that either scenario is always correct or always incorrect, just that this is a source of variance in the process.

Finally, a further related point is that even for reviewers who have reviewed the application before, their memory of what happened 8 months (the usual 2 review rounds necessary to revise) ago can vary tremendously. The official rules insist that reviewers are to destroy all information related to the review including copies of the applications and any notes made during review. I wonder how seriously people take this. I have for certain sure received summary statements of my revised applications in which substantial blocks of text are reproduced verbatim or nearly so. It takes no great leap to see that the return reviewers have held on to their prior critique document file for re-use on the revised application. Since they cannot reasonably judge which revisions they will later receive to review, it is likewise no leap to assume that reviewers may hold onto all of their reviews that they have written. Do they hold onto notes and the applications as well? Surely some of them do!

So there is clearly going to be a wide range in the familiarity a set of reviewers has with the history of, say, an A2 amended application. From a very intimate appreciation (reviewer has seen all three versions, has great memory for the review discussions and has retained the prior applications and notes) to essentially minimal historical information (a reviewer new to the panel and application). In and of itself this is bound to introduce variance into the process. As a related concern the difference in familiarity is going to bias the perceived authority of the reviewers in the impact it has on the rest of the panel. On my panel at least people are plenty forthright about saying “I’ve reviewed this one the prior two times and….” or “Well, I didn’t see the prior version but it seems responsive to me…”. This is going to have disproportional impact on the rest of the panel. As above, I’m not saying this is monolithic in direction. One might assume the clearly more-knowledgeable reviewer will seem more authoritative. However, sometimes the “more informed” reviewer comes across as the dog who will just not relinquish his/her favorite bone which the rest of the panel can see should long since have been buried.

There seems a very simple solution that would go a long ways towards reducing these sources of variance. Namely that all reviewers should be provided with all the prior versions and summary statements. This would go a long ways towards equalizing the amount of “extra” information a given reviewer brings to a revised application.