Undue influence of frequent NIH grant reviewers

February 7, 2018

A quotation

Currently 20% of researchers perform 75-90% of reviews, which is an unreasonable and unsustainable burden.

referencing this paper on peer review appeared in a blog post by Gary McDowell. It caught my eye when referenced on the twitts.

The stat is referencing manuscript / journal peer review and not the NIH grant review system but I started thinking about NIH grant review anyway. Part of this is because I recently had to re-explain one of my key beliefs about a major limitation of the NIH grant review system to someone who should know better.

NIH Grant review is an inherently conservative process.

The reason is that the vast majority of reviews of the merit of grant applications are provided by individuals who already have been chosen to serve as Principal Investigators of one or more NIH grant awards. They have had grant proposals selected as meritorious by the prior bunch of reviewers and are now are contributing strongly to the decision about the next set of proposals that will be funded.

The system is biased to select for grant applications written in a way that looks promising to people who have either been selected for writing grants in the same old way or who have been beaten into writing grants that look the same old way.

Like tends to beget like in this system. What is seen as meritorious today is likely to be very similar to what has been viewed as meritorious in the past.

This is further amplified by the social dynamics of a person who is newly asked to review grants. Most of us are very sensitive to being inexperienced, very sensitive to wanting to do a good job and feel almost entirely at sea about the process when first asked to review NIH grants. Even if we have managed to stack up 5 or 10 reviews of our proposals from that exact same study section prior to being asked to serve. This means that new reviewers are shaped even more by the culture, expectations and processes of the existing panel, which is staffed with many experienced reviewers.

So what about those experienced reviewers? And what about the number of grant applications that they review during their assigned term of 4 (3 cycles per year, please) or 6 (2 of 3 cycles per year) years of service? With about 6-10 applications to review per round this could easily be highly influential (read: one of the three primary assigned reviewers) review of 100 applications. The person has additional general influence in the panel as well, both through direct input on grants under discussion and on the general tenor and tone of the panel.

When I was placed on a study section panel for a term of service I thought the SRO told us that empaneled reviewers were not supposed to be asked for extra review duties on SEPs or as ad hoc on other panels by the rest of the SRO pool. My colleagues over the years have disabused me of the idea that this was anything more than aspirational talk from this SRO. So many empaneled reviewers are also contributing to review beyond their home review panel.

My question of the day is whether this is a good idea and whether there are ethical implications for those of us who are asked* to review NIH grants.

We all think we are great evaluators of science proposals, of course. We know best. So of course it is all right, fair and good when we choose to accept a request to review. We are virtuously helping out the system!

At what point are we contributing unduly to the inherent conservativeness of the system? We all have biases. Some are about irrelevant characteristics like the ethnicity** of the PI. Some are considered more acceptable and are about our preferences for certain areas of research, models, approaches, styles, etc. Regardless these biases are influencing our review. Our review. And one of the best ways to counter bias is the competition of competing biases. I.e., let someone else’s bias into the mix for a change, eh buddy?

I don’t have a real position on this yet. After my term of empaneled service, I accepted or rejected requests to review based on my willingness to do the work and my interest in a topic or mechanism (read: SEPs FTW). I’ve mostly kept it pretty minimal. However, I recently messed up because I had a cascade of requests last fall that sucked me in- a “normal” panel (ok, ok, I haven’t done my duty in a while), followed by a topic SEP (ok, ok I am one of a limited pool of experts I’ll do it) and then a RequestThatYouDon’tRefuse. So I’ve been doing more grant review lately than I have usually done in recent years. And I’m thinking about scope of influence on the grants that get funded.

At some point is it even ethical to keep reviewing so damn much***? Should anyone agree to serve successive 4 or 6 year terms as an empaneled reviewer? Should one say yes to every SRO request that comes along? They are going to keep asking so it is up to us to say no. And maybe to recommend the SRO ask some other person who is not on their radar?

*There are factors which enhance the SRO pool picking on the same old reviewers, btw. There’s a sort of expectation that if you have review experience you might be okay at it. I don’t know how much SROs talk to each other about prospective reviewers and their experience with the same but there must be some chit chat. “Hey, try Dr. Schmoo, she’s a great reviewer” versus “Oh, no, do not ever ask Dr. Schnortwax, he’s toxic”. There are the diversity rules that they have to follow as well- There must be diversity with respect to the geographic distribution, gender, race and ethnicity of the membership. So people that help the SROs diversity stats might be picked more often than some other people who are straight white males from the most densely packed research areas in the country working on the most common research topics using the most usual models and approaches.

**[cough]Ginther[cough, cough]

***No idea what this threshold should be, btw. But I think there is one.

18 Responses to “Undue influence of frequent NIH grant reviewers”

  1. Emaderton3 Says:

    Are you back from “sabbatical”?


  2. baltogirl Says:

    One year later and DrugMonkey returns! what have you been up to?
    I agree that if you find you are frequently reviewing the same applicants- especially negatively- you have an ethical obligation to step aside and give other viewpoints a chance, assuming the right expertise is available.
    BUT so many of us are so busy writing grants endlessly that there is no time left to review: hence the pool of willing reviewers with the correct expertise has likely shrunk.


  3. JL Says:

    Yea! The monkey!

    Now on topic: I have also felt like this about papers. Not critical, but important nonetheless. I have had to decline invitations to review papers simply because I had been revieweing so much in a given sub-sub-sub-nano-field. I wanted other biases in the pool.


  4. Morgan Price Says:

    DrugMonkey lives!
    I have to ask — why are there no post-docs reviewing grant proposals? Do you think it would be a bad idea?


  5. drugmonkey Says:

    Let’s work on getting more Asst Profs and more not-yet-funded-but-applying-like-crazy folks on panels first.


  6. odyssey Says:

    BUT so many of us are so busy writing grants endlessly that there is no time left to review: hence the pool of willing reviewers with the correct expertise has likely shrunk.

    I would counter by saying by far the best way to improve your grant writing, and to keep your finger on the pulse of what’s likely to get funded, is to review. IME it’s worth the time and effort.


  7. Cytokine Says:

    You’re back! I don’t always agree with you, but your columns offer welcomed discussion on very important topics. I sincerely hope that this is not a one-off post, but the re-emergence of your very valuable blog.

    I absolutely agree with you that we need to have more assistant professor on the panels. Grant review is a black box for most applicants and review experience is invaluable. I know of several people who chronically missed payline and once they served on a study section they better understood what reviewers were looking for and were able to get their application funded. It’s a delicate balance though, but on too many Assistant Professors and people complain that the study section lacks experience, vision, etc.

    Currently CSR rules dictate that the vast majority of the reviewers need to be Associate or above (ignoring that the criteria for moving to an associate professor position ranges wildly from institution) and the reviewers need to have had an R01. To be eligible for the Early Reviewer Program the PI can’t have an R01. That excludes a ton of highly accomplished leaders in the field who are Assistant Professors, have 1 (or more R01) and are practically ineligible to serve as a reviewer. The diversity criteria also greatly impacts who may be selected (and you’re right, some people are over selected because they fit all of the diversity requirements – minority female from the midwest. Typically they’re from small institutions).

    The reality of the situation is that in some fields (immunology, microbiology) the higher profile the person the less likely they’ll review (many (most?) of the big names haven’t reviewed in the past decade). Some fields (biophysics) have a strong culture of review and you’ll have all the big names reviewing. Unfortunately people who complain the loudest about how broken the peer review system typically (almost always) haven’t reviewed in the past decade, if at all.


  8. potnia theron Says:

    My advice (to the OldeFartes): when I am sitting on a study section, I turn down all other requests. When I am not, I tend to turn down all requests in general. I have a groups of jr to jr-ish (ie new Assoc profs) who I recommend, including POC, Women, and even one disabled person (when is the last time you had a study section with someone in a wheelchair? When you do go, do you pay attention to how accessible it is for someone who is not entirely able-bodied?).

    Interestingly, I get asked more now than when I was at MRU on east coast. Now, I count as coming from rural, and an R15 institution. The “balance” issues are real.


  9. AcademicLurker Says:

    Welcome back!

    I don’t know whether the 20/90 figure for paper reviewing that you quoted holds for NIH grant reviews, but if it does that’s a problem. Even when everyone on the study section is sincerely doing the best they can to give fair and accurate reviews (which has generally been my experience).


  10. Pinko Punko Says:

    Diversity in review critical, but should not shade review quality or expertise. Do we have expectation that some individuals are better at review than others? (I recognize super hard to quantify), but if reviewers received feedback on their written reviews and where they might improve- this might be way to raise reviewer quality across board, and would support more feeling for extending opportunity to others if there were expectation and framework to bring people up to some level quickly.


  11. qaz Says:

    Welcome back! We’ve missed you, DM!

    My understanding of this problem is not that there is a cabal of people who are trying to do all the reviews, but rather that CSR can’t get enough people to fill the gap.

    This problem came when they blocked assistant professors from the study sections. As you’ve noted in the past, that was generational politics. This has become a particularly pernicious problem as the ratio of junior to senior professors is shifting back to having a dearth of senior professors (because senior professors are now transitioning to be from genX, while the junior are the echo of the baby boomers). Maybe now that the generational politics is over, we can bring back those assistant professors. In my experience, they were often the best reviewers, the most conscientious about the work (e.g. Yun Gun), and the least likely to rush through (q.v. Horace Badger).

    As I understand the current system, assistant professors are not *allowed* to serve on study section except as this bogus “show-one-well-connected-asst-prof-the-ropes” that basically has them serve as third reviewer for a couple of proposals.

    We need a serious change that has assistant professors serving as full (if ad-hoc) members of study section.


  12. Luminiferous aether Says:

    Yeahhh The Monk is back!!!

    “We need a serious change that has assistant professors serving as full (if ad-hoc) members of study section.”

    I have seen on the SS that my apps go to, that over the past couple of rounds there have been a couple of repeat Asst. Profs as ad hocs on the panel. I may be wrong on this, but AFAIK the ECR program allows *one* opportunity for asst. profs to serve. So, I wonder if the SRO invited them again because they are good reviewers/have specific expertise that is needed/satisfy the diversity checkboxes; or because some SROs are starting to think that more Asst. Profs should be on their panels.


  13. drugmonkey Says:

    I don’t think the great Asst Prof purge was ever an absolute. Just a severe cut back from an already low number.


  14. qaz Says:

    Outside of the ECR thing, there have been no assistant professors on any study section I’ve been on(*) since the purge.

    * I consistently do about 3 panels a year. Apparently, I’m one of your 20 percent that do too many reviews. I try to mix it up by moving to panels doing different types of grants (F grants to R grants to NSF to K over the years) to fend off my impending HoraceBadgerdom as best I can.

    I can’t find any statement as to a rule. In 2015, there is a commentary response from Nakamura stating that they have about 5 percent assistant professors on study section, which implies it’s not an absolute rule.


    5 percent would be one per study section, which could be the ECRs.

    It would be good if its not a rule because then it would be possible to start bringing them back without making a major change.


  15. drugmonkey Says:

    The only review I saw published close (ca. 2004) to when Scarpa freaked out showed a max of 10% Asst Prof reviewers.

    And let us always remember that “% of reviewers” is not equal to “% of reviews” whenwe are considering the impact of Assistant Professors.


  16. girlparts Says:

    I realize it isn’t easy to get reviewers, but I’d love to see more people reviewing each grant, not just more people serving across panels. Even if just on an ad hoc basis. Almost everyone on study section seems to be doing their level best to be fair and accurate, but the process comes across as largely arbitrary for the vast middle level of grants.

    I got to do the ECR thing on a standing section, and then ad hoc on 3 special panel type things in the next year and a half. I don’t think it is a coincidence that I have gone from dozens of triaged grants to two well scored proposals in that time. It helps soooo much.


  17. drugmonkey Says:

    If the middle is indeed well populated by essentially equally meritorious proposals the outcome is not “arbitrary” surely?

    And having more reviewers to attest to the interchangeability of merit rankings wouldn’t really chance anything.


  18. Pass-through entity Says:

    Wait! What?!?

    So glad to have you back here! 🙂


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: