Merit of NIH Grants Depends on the Reviewer’s Opinion
January 27, 2021
This is not news to this audience but it bears addressing in as many ways as possible, in the context of the Hoppe et al 2019 and Ginther et al 2011 findings. Behind most of the resistance to doing anything about the funding disparity for investigators and, as we’re now finding out, topics is still some lingering idea that the NIH grant selection process is mostly about merit.
Objective merit. Sure we sort of nod that we understand that there is some wiggle room but overall it is difficult to find anyone that appears to understand something in a deep way.
“Merit” of NIH grants is untethered to anything objective. It relies on the opinion of the peer reviewers. The ~3 reviewers who are assigned to do deep review and the members of the panel (which can be 20-30ish folks) who vote scores after the discussion.
This particular dumb twitter poll shows that 77% of experienced reviewers either occasionally or regularly have the experience of thinking a grant that should not receive funding is very likely to do so.
and this other dumb little twitter poll shows that 88% of experienced NIH grant reviewers either occasionally or frequently experience a panel voting a non-fundable score for a grant they think deserves funding.
It will not escape you that individual reviewers tend to think a lot more grants should be funded than can be funded. And this shows up in the polls to some extent.
This is not high falutin’ science and it is possible we have some joker contamination here from people who are not in fact NIH review experienced.
But with that caveat, it tends to support the idea that the mere chance of which individuals are assigned to review a grant can have a major effect on merit. After all, the post-discussion scores of these individuals tends to significantly constrain the voting. But the voting is important too, since panel members can go outside the range or decide en masse to side with one or the other ends of the post-discussion range.
Swap out the assigned reviewers for a different set of three individuals and the outcomes are likely to be very different. Swap out one panel for another and the tendencies could be totally different. Is your panel heavy in those interested in sex differences and/or folks heavily on board with SABV? Or is it dominated by SABV resisters?
Is the panel super interested in the health effects of cannabis and couldn’t give a fig about methamphetamine? What do YOU think is going to come out of that panel with fundable scores?
Does the panel think any non-mammalian species is horrible for modeling human health and should really never be funded? Does the panel geek away at tractable systems and adore anything fly or worm driven and complain about the lack of manipulability available in a rat?
Of course you know this. These kinds of whines and complaints are endemic to fireside chats whenever two or more NIH grant-seeking investigators are present!
But somehow when it is a disparity of race or of topics of interest to minority communities in the US, such as from Hoppe et al 2019, then nobody is concerned. Even when there are actual data on the table showing a funding disparity. And everyone asks their “yeahbutwhatabout” questions, springing right back into the mindset that at the very core the review and selection of grants is about merit. The fact their worm grant didn’t get selected is clear evidence of a terrible bias in the NIH approach. The fact African-American PIs face a payline far lower than they do…..snore.
Because in that case it is about objective merit.
And not about the coincidence of whomever the SRO has decided should review that grant.
DrugMonkey 