Disparity of NSF funding

You are familiar with the #GintherGap, the disparity of grant award at NIH that leaves the applications with Black PIs at substantial disadvantage. Many have said from the start that it is unlikely that this is unique to the NIH and we only await similar analyses to verify that supposition.

Curiously the NSF has not, to my awareness, done any such study and released it for public consumption.

Well, a group of scientists have recently posted a preprint:

^{Chen, C. Y., Kahanamoku, S. S., Tripati, A., Alegado, R. A., Morris, V. R., Andrade, K., & Hosbey, J. (2022, July 1). Decades of systemic racial disparities in funding rates at the National Science Foundation. OSF Preprints. July 1. doi:10.31219/osf.io/xb57u.}

It reviews National Science Foundation awards (from 1996-2019) and uses demographics provided voluntarily by PIs. They found that the applicant PIs were 66% white, 3% Black, 29% Asian and below 1% for each of American Indian/Alaska Native and Native Hawaiian/Pacific Islander groups. They also found that across the reviewed years, the overall funding rate varied from 22%-34%, so the data were represented as the rate for each group relative to the average for each year. In Figure 1, reproduced below, you can see that applications with white PIs enjoy a nice consistent advantage relative to other groups and the applications with Asian PIs suffer a consistant disadvantage. The applications with Black PIs are more variable year over year but are mostly below average except for 5 years when they are right at the average. The authors note this means that in 2019, there were 798 awards with white PIs above expected value, and 460 fewer than expected awarded with Asian PIs. The size of the disparity differs slightly across the directorates of the NSF (there are seven, broken down by discipline such as Biological Sciences, Engineering, Math and Physical Sciences, Education and Human Resources, etc) but the same dis/advantage based on PI race remains.

Fig 1B from Chen et al. 2022 preprint

It gets worse. It turns out that these numbers include both Research and Non-Research (conference, training, equipment, instrumentation, exploratory) awards. Which represent 82% and 18% of awards, with the latter generally being awarded at 1.4-1.9 times the rate for Research awards in a given year. For white

Fig 3 from Chen et al 2022 preprint FY 13 – 19;
open = Non-Research, closed = Research

PI applications the two types both are funded at higher than the average rate, however significant differences emerge for Black and Asian PIs with Research awards having the lower probability of success.

So why is this the case. Well, the white PI applications get better scores from extramural reviewers. Here, I am not expert in how NSF works. A mewling newbie really. But they solicit peer reviewers which assign merit scores from 1 (Poor) to 5 (Excellent). The preprint shows the distributions of scores for FY15 and FY16 Research applications, by PI race, in Figure 5. Unsurprisingly there is a lot of overlap but the average score for white PI apps is superior to that for either Black or Asian PI apps. Interestingly, average scores are worse for Black PI apps than for Asian PI apps. Interesting because the funding disparity is larger for Asian PIs than for Black PIs. And as you can imagine, there is a relationship between score and chances of being funded but it is variable. Kind of like a Programmatic decision on exception pay or the grey zone function in NIH land. Not sure exactly how this matches up over at NSF but the first author of the preprint put me onto a 2015 FY report on the Merit Review Process that addresses this. Page 74 of the PDF (NSB-AO-206-11) has a Figure 3.2 showing the success rates by average review score and PI race. As anticipated, proposals in the 4.75 (score midpoint) bin are funded at rates of 80% or better. About 60% for the 4.25 bin, 30% for the 3.75 bin and under 10% for the 3.25 bin. Interestingly, the success rates for Black PI applications are higher than for white PI applications at the same score. The Asian PI success rates are closer to the white PI success rates but still a little bit higher, at comparable scores. So clearly something is going on with funding decision making at NSF to partially counter the poorer scores, on average, from the reviewers. The Asian PI proposals do not have as much of this advantage. This explains why the overall success rates for Black PI applications are closer to the average compared with the Asian PI apps, despite worse average scores.

Fig 5 from Chen et al 2022 preprint

One more curious factor popped out of this study. The authors, obviously, had to use only the applications for which a PI had specified their race. This was about 96% in 1999-2000 when they were able to include these data. However it was down to 90% in 2009, 86% in 2016 and then took a sharp plunge in successive years to land at 76% in 2019. The first author indicated on Twitter that this was down to 70% in 2020, the largest one year decrement. This is very curious to me. It seems obvious that PIs are doing whatever they think is going to help them get funded. For the percentage to be this large it simply has to involve large numbers of white PIs and likely Asian PIs as well. It cannot simply be Black PIs worried that racial identification will disadvantage them (a reasonable fear, given the NIH data reported in Ginther et al.) I suspect a certain type of white academic who has convinced himself (it’s usually a he) that white men are discriminated against, that the URM PIs have an easy ride to funding and the best thing for them to do is not to declare themselves white. Also another variation on the theme, the “we shouldn’t see color so I won’t give em color” type. It is hard not to note that the US has been having a more intensive discussion about systemic racial discrimination, starting somewhere around 2014 with the shooting of Michael Brown in Ferguson MO. This amped up in 2020 with the strangulation murder of George Floyd in Minneapolis. Somewhere in here, scientists finally started paying attention to the Ginther Gap. News started getting around. I think all of this is probably causally related to sharp decreases in the self-identification of race on NSF applications. Perhaps not for all the same reasons for every person or demographic. But if it is not an artifact of the grant submission system, this is the most obvious conclusion.

There is a ton of additional analysis in the preprint. Go read it. Study. Think about it.

Additional: Ginther et al. (2011) Race, ethnicity, and NIH research awards. Science, 2011 Aug 19; 333(6045):1015-9. [PubMed]

NIH data on Discussion Rates and Grant Submitting Vigor

The latest blog post over at Open Mike, from the NIH honcho of extramural grant award Mike Lauer, addresses “Discussion Rate”. This is, in his formulation, the percent of applicants (in a given Fiscal Year, FY21 in this case) who are PI on at least one application that reaches discussion. I.e., not triaged. The post presents three Tables, with this Discussion rate (and Funding rate) presented by the Sex of the PI, by race (Asian, Black, White only) or ethnicity (Hispanic or Latino vs non-Hispanic only). The tables further presented these breakdowns by Early Stage Investigator, New Investigator, At Risk and Established. At risk is a category of “researchers that received a prior substantial NIH award but, as best we can tell, will have no funding the following fiscal year if they are not successful in securing a competing award this year.” At this point you may wish to revisit an old blog post by DataHound called “Mind the Gap” which addresses the chances of regaining funding once a PI has lost all NIH grants.

I took the liberty of graphing the By-Race/Ethnicity Discussion rates, because I am a visual thinker.

There seem to be two main things that pop out. First, in the ESI category, the Discussion rate for Black PI apps is a lot lower. Which is interesting. The 60% rate for ESI might be a little odd until you remember that the burden of triage may not fall on ESI applications. At least 50% have to be discussed in each study section, small numbers in study section probably mean that on average it is more than half, and this is NIH wide data for FY 21 (5,410 ESI PIs total). Second, the NI category (New, Not Early on the chart) seems to suffer relative to the other categories.

Then I thought a bit about this per-PI Discussion rate being north of 50% for most categories. And that seemed odd to me. Then I looked at another critical column on the tables in the blog post.

The Median number of applications per applicant was…. 1. That means the mode is 1.

Wow. Just….wow.

I can maybe understand this for ESI applicants, since for many of them this will be their first grant ever submitted.

but for “At Risk”? An investigator who has experience as a PI with NIH funding, is about to have no NIH funding if a grant does not hit, and they are submitting ONE grant application per fiscal year?

I am intensely curious how this stat breaks down by deciles. How many at risk PIs are submitting only one grant proposal? Is it only about half? Two-thirds? More?

As you know, my perspective on the NIH grant getting system is that if you have only put in one grant you are not really trying. The associated implication is that any solutions to the various problems that the NIH grant award system might have that are based on someone not getting their grant after only one try are not likely to be that useful.

I just cannot make this make sense to me. Particularly if the NIH

It is slightly concerning that the NIH is now reporting on this category of investigator. Don’t get me wrong. I believe this NIH system should support a greater expectation of approximately continual funding for investigators who are funded PIs. But it absolutely cannot be 100%. What should it be? I don’t know. It’s debatable. Perhaps more importantly who should be saved? Because after all, what is the purpose of NIH reporting on this category if they do not plan to DO SOMETHING about it? By, presumably, using some sort of exception pay or policy to prevent these at risk PIs from going unfunded.

There was just such a plan bruited about for PIs funded with the ESI designation that were unable to renew or get another grant. They called them Early Established Investigators and described their plans to prioritize these apps in NOT-OD-17-101. This was shelved (NOT-OD-18-214) because “NIH’s strategy for achieving these goals has evolved based on on-going work by an Advisory Committee to the Director (ACD) Next Generation Researchers Initiative Working Group and other stakeholder feedback” and yet asserted “NIH..will use an interim strategy to consider “at risk investigators”..in its funding strategies“. In other words, people screamed bloody murder about how it was not fair to only consider “at risk” those who happened demographically to benefit from the ESI policy.

It is unclear how these “consider” decisions have been made in the subsequent interval. In a way, Program has always “considered” at risk investigators, so it is particularly unclear how this language changes anything. In the early days I had been told directly by POs that my pleas for an exception pay were not as important because “we have to take care of our long funded investigators who will otherwise be out of funding”. This sort of thing came up in study section more than once in my hearing, voiced variously as “this is the last chance for this PIs one grant” or even “the PI will be out of funding if…”. As you can imagine, at the time I was new and full of beans and found that objectionable. Now….well, I’d be happy to have those sentiments applied to me.

There is a new version of this “at risk” consideration that is tied to the new PAR-22-181 on promoting diversity. In case you are wondering why this differs from the famously rescinded NINDS NOSI, well, NIH has managed to find themselves a lawyered excuse.

Section 404M of the Public Health Service Act (added by Section 2021 in Title II, Subtitle C, of the 21st Century Cures Act, P.L. 114-255, enacted December 13, 2016), entitled, “Investing in the Next Generation of Researchers,” established the Next Generation Researchers Initiative within the Office of the NIH Director.  This initiative is intended to promote and provide opportunities for new researchers and earlier research independence, and to maintain the careers of at-risk investigators.  In particular, subsection (b) requires the Director to “Develop, modify, or prioritize policies, as needed, within the National Institutes of Health to promote opportunities for new researchers and earlier research independence, such as policies to increase opportunities for new researchers to receive funding, enhance training and mentorship programs for researchers, and enhance workforce diversity;

“enacted December 13, 2016“. So yeah, the NOSI was issued after this and they could very well have used this for cover. The NIH chose not to. Now, the NIH chooses to use this aspect of the appropriations language. And keep in mind that when Congress includes something like this NGRI in the appropriations language, NIH has requested it or accepted it or contributed to exactly how it is construed and written. So this is yet more evidence that their prior stance that the “law” or “Congress” was preventing them from acting to close the Ginther Gap was utter horseshit.

Let’s get back to “at risk” as a more explicitly expressed concern of the NIH. What will these policies mean? Well, we do know that none of this comes with any concrete detail like set aside funds (the PAR is not a PAS) or ESI-style relaxation of paylines. We do know that they do this all the damn time, under the radar. So what gives? Who is being empowered by making this “consideration” of at-risk PI applications more explicit? Who will receive exception pay grants purely because they are at risk? How many? Will it be in accordance with distance from payline? How will these “to enhance diversity” considerations be applied? How will these be balanced against regular old “our long term funded majoritarian investigator is at risk omg” sentiments in the Branches and Divisions?

This is one of the reasons I like the aforementioned Datahound analysis, because at least it gave a baseline of actual data for discussion purposes. A framework a given I or C could follow in starting to make intelligent decisions.

What is the best policy for where, who, what to pick up?