NIH Discontinues Continuous Submission for Frequent Service With A Gaslighting Excuse

January 28, 2020

The Notice NOT-ED-20-006 rescinds the continuous submission privilege for the “recent substantial service” category that has been in place since 2009 (NOT-OD-09-155). This extended the privilege that had been given to people who were serving an appointed term on a study section (NOT-OD-08-026). The qualification for “recent substantial service” meant serving as a study section member six times in an 18 month interval. In comparison, an appointed member of a study section serves in 3 meetings per year maximum, with the conversion to 6 year options entailing only two rounds per year. As a reminder the stated goal for this extension was: “to recognize outstanding review and advisory service, and to minimize disincentives to such service“. This is why it is so weird that the latest notice rescinding the policy for the “substantial service” seems to blame these people for having some sort of malign influence. To wit: “prior policy had unintended consequences, among them encouraging excessive review service and thus disproportionate influence by some.

Something smells. Really, really badly.

There is a new CSR blogpost up on Review Matters, by the current Director of CSR Noni Byrnes, which further adds to the miasma. She starts off with stuff that I agree with 1,000 percent.

The scientific peer review process benefits greatly when the study section reviewers bring not only strong scientific qualifications and expertise, but also a broad range of backgrounds and varying scientific perspectives. Bringing new viewpoints into the process replenishes and refreshes the study section, enhancing the quality of its output.

I have blogged many a word that addresses this topic in various ways. From my comments opposing the Grande Purge of Assistant Professors started by Toni Scarpa, to my comments generally on the virtues of the competition of biases to address inevitable implicit bias to my pointed comments on the Ginther finding and NIH’s dismal response to same. I agree that broadening the participation in NIH peer review is a good goal. And I welcome this post because it gives us some interesting data, new to my eyes.

As of January 1, 2020, there were 22,608 individuals with active R01 funding. Of these, 30% (6715) have served one to five times, and 18% (4074) have never served as a reviewer in the last 12 years. Of those who have served only one to five times over 12 years, 26% are assistant professors and 34% are associate professors.

Cool, cool. At least it is a starting point for discussion. Should they be trying to reduce that 18% number? Heck yes. To what? I don’t know. Some of this is structural in the sense that someone just awarded their first R01 probably is less likely to have a service record within the next 3 months. Right? So…5%? The question is how to do this, why are the 18% being overlooked, etc. Well, if you are the head of CSR you know in your bones that peer review service is opt-in…but only opt-in upon request from a CSR (or in limited cases IC-associated) SRO. So the 18% needs to be parsed into those who have never been asked and those who have refused. Those that have been asked several times (3+ over time?) and those that have only been asked once (I mean, stuff happens and you aren’t always available when requested). And Director Byrnes is sorta, half heartedly, putting the blame where it belongs, pending data on refusal rates, on the SROs. “ In an effort to facilitate broader participation in review, we are making these data available to SROs and encouraging them to identify qualified and scientifically appropriate reviewers, who may not have been on their radar previously. ” “Encouraging” . Gee, for some reason, the SROs I talked to during the Scarpa Grande Purge suggested that he was doing a lot more than mere “encouragement” to get rid of Assistant Professors. And in full disclosure more than one SRO alluded to fighting back and slow-walking since they disagreed with Scarpa’s agenda. But both of these things suggest that Byrnes is going to have to do more than just show her SROs the data and ask them nicely to do better.

Then the blog post goes into a topic I think I’ve planned to blog, and failed to do so, for years. Disproportionate influence of a given reviewer, by virtue of constant and substantial participation in peer review of grants. This is a tricky topic. As I said, the system is opt-in upon request. So a given reviewer is at least partially to blame for the number of panels he or she or they serve on. The blog post has a nice little graph of distribution of the 12 year service history of anyone who has been on a CSR panel in the past two years.

However, one aspect of broadening the pool of reviewers is to avoid excessive review service by a small fraction of people, which can lead them to have a disproportionate effect on review outcomes. We are looking into issue of undue influence, or the “gatekeeper” phenomenon, where a reviewer has participated in the NIH peer review process at a rate much higher than their peers, and thus has had a disproportionate effect on review outcomes in a given field.

Look, Dear Reader, by now you know what the primary analysis from the peanut gallery will be. If you think a given reviewer hates your work, your approaches, you, your pubs, etc, you think they are having undue influence on the study section to which you are submitting your grants. It is particularly inflaming when you can’t seem to escape them because no matter whether you send stuff to your best fit study section, various SEPs, try a different mechanism, etc….up they pop. Professor Antagonist the Perma-Reviewer. On the other hand, if a reviewer that you think is sympathetic to your proposals keeps showing up, heck you wouldn’t complain if that continued on 75% of the sections your grants are reviewed in for decades. Right?

Director Byrnes drew her first line on the chart at 1-36 meetings per 12 year interval. Now me, I think I want to see something a little bit closer and more segmented on that. Three meetings, year in, year out for 12 years does seem like a fairly substantial and outsized influence. One per year does not. One 12 round interval of service (three rounds per year in 4 years or two in 6) as an appointed reviewer seems okay to me. The chart then shows quite a number of people in the 37-72 meeting range (5% of the sample) and even some folks in the 73+ range (1%ish). The way they are talking about undue influence it seems like they should be in the low single digits, right?

But they are not. The minimum standard was 6 review panels per 5 rounds. This is one more than is standard for an empaneled reviewer. And she or he could always just pick up an extra one, right? And I went to view the video of the Advisory Council meeting linked in the blog post and there is a suggestion that the real problem are reviewers begging SROs for an assignment at the last minute to keep their eligibility. Right? So they are for sure pointing the finger at people who meet the bare minimum. Which is not much different from that “influence” wielded by a term of service.

And probably even less. Why? because if you are cobbling your 6 out of 5 rounds from ad hoc requests it is very likely to be entailing a smaller load. SEP service, in my experience, means a smaller review load per panel. So does ad hoc service on established panels, frequently, because the SRO is trying not to annoy the ad hocs and the empaneled folks have buy in. The Advisory Council discussion came oh so close to covering this but veered away into distraction. In part because Director Byrnes started talking about voting scores as being more important than reviews written….but this is also correlated. For the SEPs, fewer items per reviewer often comes with a smaller overall panel load compared with a standing panel. I’ve been on established study sections that routinely have anywhere from 60-90 apps per round. Rarely, if ever, on a SEP with more than about 30.

I really don’t understand the CSR logic here.

The only thing that makes any sense is that they are tired of having to route so many last-minute applications once SROs of standing panels have started trying to assign apps and recruit ad hoc members. And maybe tired of having to convene 5-15 app SEPs to deal with the overflow. Certainly my personal experience has been that in the past few years my continuous submission go to SEPs and are refused by the standing panel SROs. This never used to happen to me in the first years of this policy.

but who knows.

10 Responses to “NIH Discontinues Continuous Submission for Frequent Service With A Gaslighting Excuse”

  1. Ola Says:

    There are lots of issues at CSR, for sure, but I don’t really see “over-service” of reviewers as an urgent thing that demands their full attention. Just quietly deal with it via the SROs, and maintain an internal list of reviewers who should not be assigned more service. As I think you’re implying, focus on issues like this results in gaslighting on the bigger issues at hand such as Ginther. Undue influence is caused by so many factors that have nothing to do with amount of service – having the loudest voice in the room, being a “Columbo” (just one more thing), voting far outside the range on every. fucking. proposal., serving on SEPs, etc.


  2. Joe Says:

    Serving on study section is a lot of work for very little reward, at least once you’ve done it awhile. If they are getting rid of continuous submission, they need to come up with other, better rewards for service.


  3. drugmonkey Says:

    “Colombo”. Had not heard that one. Accurate.

    Yeah I think this is a weird policy change that isn’t aligned with the stated goal. This will affect a lot more than a consensus of over active reviewers. And, as you say, only address undue influence very imprecisely.


  4. Science Geek Says:

    Lay off the SROs. If you think this is gaslighting try being an SRO. That’s every day in the life of an SRO, gaslit by supervisors. Now they discovered overuse? All these years they kept telling SROs not to recruit junior PIs, not to recruit asst profs, limit the associates. Then suddenly the full profs are overused. And now we decided they will diversify. Also, the 18% is very vague. How many are: 1. retired?; 2. left academia? 3. sadly passed away? 4. perpetually say no? and the questions go on.


  5. drugmonkey Says:

    Yeah, I know SROs are in a bad position with respect to systemic change. But they have *some* role. And I am certain that they vary in their stances. I know some personally who tried to fight the Scarpa purge of Assistant Professors as long as they could. Some were obviously fighting for (and against) the idea that the ECR program was for underrepresented individuals. Etc.

    And yes, I imagine they are on tight deadlines to recruit warm bodies who will review grants and that is not necessarily good for seeking outside of the box.

    Yes it is hard. Yes there are excuses.

    Just like all of us have to averting our eyes from anything that is bad and harmful about our respective professions.

    I understand their situation but I am not letting them off the hook just because they work in an enormous bureaucracy.


  6. Science Geek Says:

    Sure there are excuses but the Scarpa era is long gone. There was the Nakamura era and now the Byrnes era.
    I disagree about “not thinking outside of the box”. Most SROs do. But in the Nakamura era (Byrnes era is too short to evaluate) if an SRO had a panel of majority junior PIs, no matter how good the credentials were, they would be slammed by their bosses. How? 1. Ask to remove people from the panel and get senior PIs; 2. Let the review go and if one applicant complained (especially if the applicant is a heavyweight) or PO and then all mayhem breaks loose; 3. When staff evals come and the metrics appear.

    Now the ECR program was not done right and it is chaotic. If they are serious they should remove all restrictions and allow qualified reviewers. Now you will say what merits qualified? IMHO if you are an Asst Prof or equivalent, have an independent research program and pubs. If you have submitted some form of an NIH grant it’s a plus but not a dealbreaker. Even better 25% of panels should have new investigators (the ones without an R01 yet).

    So some do continue to fight and every review round they put up with version 1-3. They don’t last long.

    What they need is leadership thinking outside the box. But….


  7. drugmonkey Says:

    Sure there are excuses but the Scarpa era is long gone. There was the Nakamura era and now the Byrnes era.
    Pretending that things have changed just because the people have is ridiculous. As is thinking what happened in the past can be safely ignored. History is essential to true understanding of everything.

    But in the Nakamura era (Byrnes era is too short to evaluate) if an SRO had a panel of majority junior PIs, no matter how good the credentials were, they would be slammed by their bosses.

    So that’s two eras with the same policy. What makes you think Byrnes will change anything if we do not demand it and do not demand they account for themselves as an institution for past actions. Did anything get better by purging the junior folks? No, it got worse.

    Now the ECR program was not done right and it is chaotic.
    It is essential to understand why it was not done right and why it is chaotic if we are to fix the many ills of review. You can’t just box it up and say “let’s do something else” if you don’t understand how and why that program got kneecapped from the origin.

    What they need is leadership thinking outside the box.
    They also need leadership being told by applicants that this is unacceptable until they finally start listening to something like the whole distribution, not just the old and very well funded elites.


  8. Science Geek Says:

    I am with you I just think is not fair to get on the SROs case because most of them will agree with you. Even if they want to change things they can’t. The calls are made in the front office and the rank and file are told to obey. The only difference was that in the Nakamura era, top brass did listen on occasion.

    Byrnes will not change anything, I agree. Demands from the junior PI community will not do much either. They will be dismissed as complaints. And if you say things got worse, they will throw a bunch of metrics and say “not really”.

    An example of NOT having junior PI input: the ECR program was recently “improved” by having two ECRs on the panel but their assignments cut down by half. This “improvement” was done mainly with senior PI input (see last September’s council). So the bone to the junior PIs was the increase in number but what are the odds an ECR will get to have an app discussed when they are assigned two? Another gem while we are at it. How many new PIs are contributing to the enquire program?

    By outside the box, that’s what I meant. Listen and respond to the greater applicant community. Not the established PIs.

    Also another problem is that NIH leadership thinks CSR is doing a splendid job. So things are not looking good.

    So nothing much will change.


  9. drugmonkey Says:

    the ECR program was recently “improved” by having two ECRs on the panel but their assignments cut down by half. This “improvement” was done mainly with senior PI input (see last September’s council).

    Ugh. Terrible. I still say the ECRs should get a moderate load. Like maybe 5 apps. And at least one as primary. Not this training wheels, thin cover for statutory obligation, observer status.


  10. Science Geek Says:

    “Not this training wheels, thin cover for statutory obligation, observer status.”

    I could not have said it any better. To the outside observer, it is two ECRs and CSR presents this as progress, inclusion of new PIs to the review process and so on.

    The CSR leadership cares more about appearances and checking the boxes than substance. And also appeasing the staunch defenders of the status quo.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: