The burden of triage should not fall unfairly upon the ESI/NI

I have heard a very dispiriting rumour floating about and I raise it on the blog to see if any of my Readers have seen similar things happening.

Once upon a time, the NIH came to the realization that the peer review process for grant applications had a bias against the less-established, newer, younger, etc Principal Investigator. That is, their proposals did not score as well and were not being funded at the same rate as those applications on which more senior and established investigators were the PI.

Someone clearly came to the conclusion, which I share, that this difference was not due to any meaningful difference in the chance that the ensuing science would be valuable and productive. So the NIH set about a number of steps to redress the situation.

One of the pathetic bandaid solutions steps they came up with was to ensure that the burden of triage did not fall disproportionally upon the younger PI applications.

As you are aware, approximately half of applications do not get discussed at the meeting. This is based on preliminary scores issued by the three assigned reviewers, generated prior to the actual meeting date. Without being fully considered by the entire committee during discussion, an application cannot be “rescued” from various sources of unfair or bad review. Although we all recognized that a full rescue from nearly-triaged to a fundable score is rare it is at least possible. And since the NIH is really just looking at aggregate scores when it comes to the bias-against-noobPI-apps stuff, the movement in the positive direction is still a desired goal.

What someone in the halls of the Center for Scientific Review at the NIH realized was that if noobPI apps were generally scoring worse than those of established PIs, then they were more likely to be triaged. So if the general triage line was 50% of applications, then perhaps 75% of Early Stage Investigator or New Investigator apps were being triaged.

The solution was to put down an edict to the SROs that “the burden of triage should not fall unfairly upon the ESI/NI applications”. Meaning that when the triage lines were originally drawn based on the preliminary scores, the SRO had to specifically review the population of ESI/NI apps and make sure that an equal proportion of them were being discussed, let’s say 50% for convenience. This meant that sometimes ESI apps were being dragged up for discussion with preliminary scores that were worse than scores of several apps from established PIs which were being triaged/not discussed.

You will anticipate my skepticism. At the time, and this was years ago by now, I thought it was a ridiculous and useless move. Because once the preliminary scores were in that range, they were very unlikely to move. And it did nothing to address the presumed bias that put those ESI/NI scores so much lower, on average, then they should have been. It was a silly dodge to keep the aggregate numbers up without doing anything about the fundamental outcome- fundable? or not-fundable?

HOWEVER.

The rumour I have heard is that some SROs have been interpreting this rule to mean that only 50% of ESI/NI apps should be discussed. A critical distinction from “at least“, which was my prior understanding of the policy and certainly how it was used in any study sections I participated in. In such a new interpretation of the policy, there would potentially be established investigator applications being discussed which had preliminary scores worse than some ESI applications that were not being discussed.

This is so backwards that it burns.