NIH Discontinues Continuous Submission for Frequent Service With A Gaslighting Excuse

The Notice NOT-ED-20-006 rescinds the continuous submission privilege for the “recent substantial service” category that has been in place since 2009 (NOT-OD-09-155). This extended the privilege that had been given to people who were serving an appointed term on a study section (NOT-OD-08-026). The qualification for “recent substantial service” meant serving as a study section member six times in an 18 month interval. In comparison, an appointed member of a study section serves in 3 meetings per year maximum, with the conversion to 6 year options entailing only two rounds per year. As a reminder the stated goal for this extension was: “to recognize outstanding review and advisory service, and to minimize disincentives to such service“. This is why it is so weird that the latest notice rescinding the policy for the “substantial service” seems to blame these people for having some sort of malign influence. To wit: “prior policy had unintended consequences, among them encouraging excessive review service and thus disproportionate influence by some.“

Something smells. Really, really badly.

There is a new CSR blogpost up on Review Matters, by the current Director of CSR Noni Byrnes, which further adds to the miasma. She starts off with stuff that I agree with 1,000 percent.

The scientific peer review process benefits greatly when the study section reviewers bring not only strong scientific qualifications and expertise, but also a broad range of backgrounds and varying scientific perspectives. Bringing new viewpoints into the process replenishes and refreshes the study section, enhancing the quality of its output.

I have blogged many a word that addresses this topic in various ways. From my comments opposing the Grande Purge of Assistant Professors started by Toni Scarpa, to my comments generally on the virtues of the competition of biases to address inevitable implicit bias to my pointed comments on the Ginther finding and NIH’s dismal response to same. I agree that broadening the participation in NIH peer review is a good goal. And I welcome this post because it gives us some interesting data, new to my eyes.

As of January 1, 2020, there were 22,608 individuals with active R01 funding. Of these, 30% (6715) have served one to five times, and 18% (4074) have never served as a reviewer in the last 12 years. Of those who have served only one to five times over 12 years, 26% are assistant professors and 34% are associate professors.

Cool, cool. At least it is a starting point for discussion. Should they be trying to reduce that 18% number? Heck yes. To what? I don’t know. Some of this is structural in the sense that someone just awarded their first R01 probably is less likely to have a service record within the next 3 months. Right? So…5%? The question is how to do this, why are the 18% being overlooked, etc. Well, if you are the head of CSR you know in your bones that peer review service is opt-in…but only opt-in upon request from a CSR (or in limited cases IC-associated) SRO. So the 18% needs to be parsed into those who have never been asked and those who have refused. Those that have been asked several times (3+ over time?) and those that have only been asked once (I mean, stuff happens and you aren’t always available when requested). And Director Byrnes is sorta, half heartedly, putting the blame where it belongs, pending data on refusal rates, on the SROs. “ In an effort to facilitate broader participation in review, we are making these data available to SROs and encouraging them to identify qualified and scientifically appropriate reviewers, who may not have been on their radar previously. ” “Encouraging” . Gee, for some reason, the SROs I talked to during the Scarpa Grande Purge suggested that he was doing a lot more than mere “encouragement” to get rid of Assistant Professors. And in full disclosure more than one SRO alluded to fighting back and slow-walking since they disagreed with Scarpa’s agenda. But both of these things suggest that Byrnes is going to have to do more than just show her SROs the data and ask them nicely to do better.

Then the blog post goes into a topic I think I’ve planned to blog, and failed to do so, for years. Disproportionate influence of a given reviewer, by virtue of constant and substantial participation in peer review of grants. This is a tricky topic. As I said, the system is opt-in upon request. So a given reviewer is at least partially to blame for the number of panels he or she or they serve on. The blog post has a nice little graph of distribution of the 12 year service history of anyone who has been on a CSR panel in the past two years.

However, one aspect of broadening the pool of reviewers is to avoid excessive review service by a small fraction of people, which can lead them to have a disproportionate effect on review outcomes. We are looking into issue of undue influence, or the “gatekeeper” phenomenon, where a reviewer has participated in the NIH peer review process at a rate much higher than their peers, and thus has had a disproportionate effect on review outcomes in a given field.

Look, Dear Reader, by now you know what the primary analysis from the peanut gallery will be. If you think a given reviewer hates your work, your approaches, you, your pubs, etc, you think they are having undue influence on the study section to which you are submitting your grants. It is particularly inflaming when you can’t seem to escape them because no matter whether you send stuff to your best fit study section, various SEPs, try a different mechanism, etc….up they pop. Professor Antagonist the Perma-Reviewer. On the other hand, if a reviewer that you think is sympathetic to your proposals keeps showing up, heck you wouldn’t complain if that continued on 75% of the sections your grants are reviewed in for decades. Right?

Director Byrnes drew her first line on the chart at 1-36 meetings per 12 year interval. Now me, I think I want to see something a little bit closer and more segmented on that. Three meetings, year in, year out for 12 years does seem like a fairly substantial and outsized influence. One per year does not. One 12 round interval of service (three rounds per year in 4 years or two in 6) as an appointed reviewer seems okay to me. The chart then shows quite a number of people in the 37-72 meeting range (5% of the sample) and even some folks in the 73+ range (1%ish). The way they are talking about undue influence it seems like they should be in the low single digits, right?

But they are not. The minimum standard was 6 review panels per 5 rounds. This is one more than is standard for an empaneled reviewer. And she or he could always just pick up an extra one, right? And I went to view the video of the Advisory Council meeting linked in the blog post and there is a suggestion that the real problem are reviewers begging SROs for an assignment at the last minute to keep their eligibility. Right? So they are for sure pointing the finger at people who meet the bare minimum. Which is not much different from that “influence” wielded by a term of service.

And probably even less. Why? because if you are cobbling your 6 out of 5 rounds from ad hoc requests it is very likely to be entailing a smaller load. SEP service, in my experience, means a smaller review load per panel. So does ad hoc service on established panels, frequently, because the SRO is trying not to annoy the ad hocs and the empaneled folks have buy in. The Advisory Council discussion came oh so close to covering this but veered away into distraction. In part because Director Byrnes started talking about voting scores as being more important than reviews written….but this is also correlated. For the SEPs, fewer items per reviewer often comes with a smaller overall panel load compared with a standing panel. I’ve been on established study sections that routinely have anywhere from 60-90 apps per round. Rarely, if ever, on a SEP with more than about 30.

I really don’t understand the CSR logic here.

The only thing that makes any sense is that they are tired of having to route so many last-minute applications once SROs of standing panels have started trying to assign apps and recruit ad hoc members. And maybe tired of having to convene 5-15 app SEPs to deal with the overflow. Certainly my personal experience has been that in the past few years my continuous submission go to SEPs and are refused by the standing panel SROs. This never used to happen to me in the first years of this policy.

but who knows.