Your Grant In Review: Junior Reviewers Are Too Focused on Details

Although the enthusiastic witch hunt against the mythical Assistant Professor on study section who “killed my grant” has been percolating along for several years now, I have yet to come across a well supported expression of the complaint. I hear a lot of mewling about how we need “better” and “more experienced” reviewers in the context of calls for reviewers of more advanced age and career status. What I do not hear is a lot of explicit and well-reasoned argument for why older reviewers are necessarily better and why less experienced reviewers are necessarily worse.
When I can pin down someone in person, however, I can extract a limping argument so I might as well address that.

The argument against the more-junior reviewers on NIH study section review that is the most amenable to discussion is that the younger people are unable to see the forest for the trees. Unable to take the broad view. Have a tendency to focus on details and grantsmanship issues in preference to the quality of the science (wot, wot?). I will note the evidence for this is quite lame, as there are never any studies or CSR datasets being referenced. What I do hear generally follows the following trends.

1. The comically flawed assumption that the PI in question knows which of the critiques of their grants were written by less-tried investigators.
2. The perspective of the PI’s experience with the (1?, 3?) Assistant Professors in their Department who give them informal comments on their own grant.
3. Occasionally some direct study section experiences in which it is claimed that the Assistant Professors focus on detail too narrowly.

For argument’s sake, let us credit these observations and ask why this might be the case.
My first thought is that the nit-picky, comprehensive review approach has a lot to do with the Assistant Professor not wanting to look like a fool in the study section meeting. This is perfectly natural. Nobody wants to look an idiot in any scientific setting and the study section is one place where you are expected to be reaching a bit outside of your most comfortable domains. It can be terrifying. I was VERY nervous on my first invitation to ad hoc review. Particularly since I was visiting a panel which had, on multiple opportunities to review my grant proposals, essentially told me they already thought I was an idiot* (that’s what triage means, right?)! I hear this from colleagues on their first appearances as well and let us recognize that I am not a particularly wilting violet. So I tend to assume this is a consistent feature. Assistant Professors go into exhaustive detail to avoid looking foolish in front of their peers.
I would argue that this has less to do with their career status and more to do with their specific experience reviewing grants. A few meetings under the belt and I wouldn’t be surprised if this tendency (if generally true) is much reduced. Score one for experience over career tenure.
My second thought has to do with the way that these Assistant Professors have been taught to review grants- by way of reading the summary statements provided for their own submissions! Right? On our study section at any rate it is very common that the pool of newbie reviewers is drawn from the pool of recent applicants. So if the reviewer has a handful of summary statements from the BunnyHopper panel and then goes to review grants in that panel, might she not conclude that the proper way to review grants is the way that her applications have been reviewed?
And, of course, since Assistant Professor PIs have been disproportionately criticized for things like publication output (without regard for the fact they have been building a lab on one R21 and startup instead of operating on 3 R01s), cleanliness of grant preparation, clarity of hypotheses, interpretation of results and alternative procedures, relevance of background, relevance and abundance of preliminary data, proof of familiarity and function of the methods proposed (you know, StockCritiques)….is it any wonder they direct the same critiques at the applications they review?
They may do so for two reasons. First and foremost because they have been trained to think this is the way review is supposed to go. After a few rounds of study section, well, such a reviewer might start to notice that certain obvious StockCritique bait is not inevitably snapped up. Reviewers choose to overlook much of that stuff–completely coincidentally, I am sure, on the more established PIs grants so how would our Assistant Professor protagonist know that in advance? Again, we can score one for direct experience reviewing and zero for career status.
The second reason has to do with fairness. Once the n00b-ish reviewer has a little experience with the way study section goes and realizes that senior investigator applications get passes for stuff that they had been beaten up for repeatedly, and that younger investigators generally get hit for, well, they get a little pissed. [Are you recognizing anyone here, DearReader?] And they think to themselves that they are for damnsure not going to perpetuate the status quo- what is sauce for the goose is sauce for the gander and all that. You can’t really argue with fairness, can you? I mean the explicit CSR instructions touch on fairness and equality of review in many ways. We are in fact supposed to focus only on the merits of the application as presented and they tell us so repeatedly, even in the course of discussion (if the SRO is attentive, anyway).
So what’s the big deal then? Why would senior investigators be complaining if their grants are reviewed the way junior investigators’ grants are reviewed? Why would they be insisting this shows that there is a Problem with review? Why would the supposed problem be with the junior reviewer instead of the more-senior reviewer who gives the senior PI a pass on things that the junior PI is beaten up over?
Why indeed?
__
*honestly, they were very welcoming and I got a few of the “I’m really glad you finally got your grant” type comments.