NINDS joins the grant-funding-outcome party

First it was NIAMS which joined NIGMS in publishing funding outcome data that is of high interest to applicants. I’m referring to the number of grants funded/not funded by priority score. NIGMS has presented these data for ages and the National Institute of Arthritis and Musculoskeletal and Skin Diseasesput theirs up for the first time just this summer.

The new data from NINDS for Fiscal Year 2011 are remarkably consistent with the datasets posted by NIGMS and by NIAMS.

This graph presents the R01 data, inclusive of new and competing continuation applications and of both experienced and New Investigators/Early Stage Investigators. You can visit their page for further graphical breakdowns.

What is readily apparent from this graph, as with all similar graphs that I’ve seen to date, is that there is a readily perceptible percentile-rank payline under which essentially all grants are funded. Above this apparent payline, funding is possible but comparatively rare. Furthermore, in this zone of exception funding (aka grey zone, aka “pickups”) the chances of getting funded are best for apps with scores closest to the apparent payline. The probability of the exception funding decreases as a function of this distance.

This tells you, as always, that the major input to the system continues to be the score ranks (and therefore percentiles) decided by the study section review. The relative impact of Program decisions to fund grants out of order is comparatively small. In the broader scheme.

But yes, if you are one of those with an application in the 15-18%ile that didn’t get funded by NINDS while they were picking up 25+%ile scores this is a HUGE impact. I’ve been on both sides of this fence (I assume) in the past so….yeah.

The rather interesting bits in all of this have to do with my comment about “remarkably consistent” and “apparent payline”. As you know, some of the NIH ICs publish their paylines formally and some do not. In fact, many of the latter insist that they do not in fact have a payline. As it happens, most ICs from whom I tend to seek money fall into the latter category. What you realize over the years of calling Program Officers to beg for some small indication of your grant’s chances of funding is that they all have paylines. There is always some internal sense of what kind of score is going to be a near-certainty, what is a “no-way” and what might be arguable depending on the final details. What you also gain a vague picture of, through this and through talking to your peers about what funded and what did not, a distribution much like the above figure. So, as we start to see more and more ICs (with different alleged payline policies) post their data, we can see that they all behave more or less the same when it comes to funding grants by initial review scores/percentiles.

There are also certain ICs that bruit it about that they stick strictly to the order of peer review. I.e., that they never skip a grant under the payline and that, more importantly, that they never fund differentially above the payline. They imply quite strongly, or state outright, that grants with equal ranks above the payline will all be funded or not funded.

I have been immensely skeptical of such claims and insisted that I wouldn’t believe it until I see the funding data.

NINDS is one of these ICs that claims/is claimed to fund in “strict order”.

These data show that is totally false.

I LOVE being right.

__
[h/t: PhysioProf]