“Merit” and the NIH disparity of grant award to Black PIs

June 10, 2020

I was just noticing something that I hadn’t really focused on before in the Hoppe et al 2019 report on the success of grant applications based on topic choices. This is on me because I’d done an entire blog post on a similar feature of this situation back when Ginther et al 2011 emerged. The earlier blog post focused on the quite well established reality that almost all apps are funded up to a payline (or virtual payline for ICs that claim, disingenuously, that they don’t have one) and that the odds of being funded as one moves away from (worse scoring) that payline, the lower the odds. Supplemental Figure S1 in Ginther showed that these general trends were true for all racial groups.

My blog post was essentially focused on the idea that some apps from African-American PIs were not being funded at a given near-miss score while some apps from white PIs were being funded at worse scores.

It’s worth taking a look at this in Hoppe et al. because it is a more recent dataset from applications scored using the new nine point scale.

I was alerted to Table 1 of Hoppe et al. which shows the percentage of the total funded pool of applications from Black and white PIs by the voted percentile rank, binned into 5 percentile ranges (0-4 is good, 85-89%ile bad).

As you would expect, almost all applications in the top two bins (0-9%ile) were funded regardless of PI race. And the chances of an app being funded at a given percentile bin decrease the further they are away from the very top scores. Where it gets interesting is after the 34%ile mark where no Black PI apps were funded. In any score bin. And there was at least one application in each bin save for 65-69, 75-79 and 80-84 which are not worth talking about anyway.

The pinch is observing that at least some applications of white PIs were funded from 35-59th percentile. I.e., at scores that are worse than the score of any app funded with a Black PI. On Twitter I originally screwed up the count because I stupidly applied the bin percentages to the entire population of funded awards. Not so. In fact I need to calculate it per bin.

Now if my current thinking is right, and it may not be, those bonus bins for white PIs represent 25% of the distribution (5 bins, 5%ile points per bin). The supplement Table S1 tells us there were 103,620 applications submitted by white PIs so that leaves us with 25,905 applications, 5,181 in each bin.

This is very rough.

Percentiling of applications is within a rolling three rounds of each standing study section. Special Emphasis Panels are variously percentiled- sometimes against an associated parent study section, sometimes against the total CSR pool.

But let’s take this as the aggregate for discussion.

Multiplying each of the bin success rates, I end up with a total of 119 applications of white PIs funded from 34-59th percentile. A score range at which ZERO applications were funded to Black PIs.

So, in essence, you could replace all of those applications funded to white PIs with more meritorious (well? that’s how they use the rankings. percentile = merit) unfunded applications submitted by Black PIs. Even by some distance as only 74% of 10-14%ile scoring applications with Black PIs were funded for example.

I was curious why Hoppe et al included the Table and what use they made of it. I could find only one mention of Table 1 and it was in the section titled “IC decisions do not contribute to funding gap“.

However, below the 15th percentile, there was no difference in the average rate at which ICs funded each group (Table 1); applications from AA/B and WH scientists that scored in the 15th to 24th percentile range, which was just above the nominal payline for FY 2011–2015, were funded at similar rates (AA/B 25.2% versus WH 26.6%, P = 0.76; Table 1). The differences we observe at narrower percentile ranges (15 to 19, 20 to 24, 25 to 29, and 30 to 34) slightly favored either AA/B or WH applicants alternately but were in no case statistically significant (P ≥ 0.13 for all ranges). These results suggest that final funding decisions by ICs, whether based on impact scores or discretionary funding decisions, do not contribute to the funding gap.

This is more than a little annoying. Sure, they sliced and diced the analysis down to where it is not statistically resolvable as a difference. But real world? It’s not a matter of constant anger for any PI who has a near miss score and gets wind of anyone being funded at a worse score? Sure it is.

And that last statement is just plain false. 119 white PI applications funded at worse scores is 46.5% of the total number of applications funded with Black PIs. If all of those discretionary funding decisions had gone to Black PIs, that would raise the hit rate from 10.7% to 15.6% for Black PIs. Whereas the white PI hit rate would plunge from 17.7% to…17.56%.

So this analysis they are referring to supports quite the opposite conclusion. Discretionary funding decisions, i.e. outside of percentile ranks where nearly every application is funded, do in fact contribute substantially to the disparity.

and correcting this to give Black PIs a fair hit rate, by selecting applications of HIGHER MERIT, would cause an entirely imperceptible change in the chances for white PIs.

4 Responses to ““Merit” and the NIH disparity of grant award to Black PIs”

  1. Grumpy Says:

    Any interest in doing a post on this open Mike blog update: “Anonymizing Peer Review for the NIH Director’s Transformative Research Award Applications” ?

    Common Find decided that double blind review is the solution to their problems of disparity in outcomes based on demographics and geography/institution for HRHR programs. The data showing just how bad their problem is can be found in recent HRHR Advisory Committee reports, https://commonfund.nih.gov/newinnovator/programevaluation.

    Comments are filled with praise and re-affirmations of the goal of Objective Scientific Merit and funding The Best Science.


  2. Arlenna Says:

    This is really important DM. I am in a situation where my grant was being considered outside the NCI payline, but it will ultimately not be funded. This sucks a lot, yes. But I am white and can submit more grants and will benefit from the privilege of an enhanced likelihood to eventually get something funded. Based on the data here, I would be pretty surprised if the NCI leadership decided to fund an outside-of-payline grant from a Black PI in my place–however, I would be really glad if that was the case.

    I’m not asking for a pat on the back, but am just communicating to other white PIs, that we need to be okay with this and be willing to be the trade-offs in these hard decisions outside the payline, in favor of Black PIs. We have gotta get past this scarcity protection of our privilege and just suck it up, even if it hurts our careers in the short term or even the long term. As you and others have been pointing out for so long, and as these numbers show pretty clearly, it is messed up that our white success is predicated on the systematic devaluing of Black scientists’ contributions. This selective bias advantage being conferred on white PIs by IC decisions needs to stop.


  3. drugmonkey Says:

    Grumpy- I’ve gone on and on about how blinded peer review isn’t going to actually work and how we need to hold feet to the fire for them to prove that it has. I’m not gearing up for that again right now.
    Arlenna- we’re not in the space of asking individual PI’s “hey would you give up your grant” , in my opinion. There are so few on the basis of an IC funding round decision that they just exist in the space of all the other bubble scores that may or may not be funded. ICs just have to change their prioritization calculus. 17.7 vs 17.5% hit rate? nobody is going to even notice that.


  4. Grumpy Says:

    One easy babystep for NIH to address the funding disparity would be to announce a PAR for P01/P20 type grants exclusively for HBCUs and HSIs. Not just training or transition grants but real R01-level funding for a cohort of faculty at a few institutions. Other federal funding agencies have programs like that, why doesn’t NIH? It’s no replacement for the no-brainer R01 payline bumps you are proposing, but it should be an easy thing to do that is right in line with analogous DoD/DOE/NSF programs.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: