Your Grant in Review: Impacts of the New Scoring System

Having completed my first go-round with the new NIH scoring system earlier in the month, I’ve been trying to reflect on the good, bad and ugly. (It is still very early days on this but discussions are starting to emerge.)
I think my biggest surprise and resulting problem has to do with the fact that the actual study section discussion is supposed to follow the new critique format. The emphasis of each reviewer’s presentation is supposed to start with the overall impact, move on to significance, then investigators…etc. The detailed discussion of the approach (so familiar to the prior way of doing business) is, let us say, heavily discouraged. So the rest of the panel, who for the most part haven’t actually read the application being discussed, have no idea what the proposal is about. From chatting with other study section members who have been participating in these new review meetings I think that “frustrated” is the nicest way to describe the reaction.
Now, I am not saying that every grant received vigorous input from the entire panel of people. But at least in times past, I’d say a majority of the discussions got at least one substantive comment from another panel member, not directly assigned to the application. I’d hate to put a number on it but many applications in each and every round had a vigorous multi-reviewer discussion. So far it looks as if the participation of the other reviewers is sharply reduced. We completed our meeting earlier in the month in record time, for example.
Ideally, I would hope that reviewers will adjust to this new approach in the future by doing a little be more preparation in the week leading up to the meeting. A little more reading of the other proposals that appear likely to be discussed. And I do like the fact that the focus is supposed to be moved away from whether the minutia of the methodological approach satisfy the whims of a given reviewer. Nevertheless I suppose I didn’t anticipate the degree to which not talking about the approach left the unassigned reviewers in the dark and feeling as if they could not supply meaningful input to the discussion.