Disparity of NSF funding

You are familiar with the #GintherGap, the disparity of grant award at NIH that leaves the applications with Black PIs at substantial disadvantage. Many have said from the start that it is unlikely that this is unique to the NIH and we only await similar analyses to verify that supposition.

Curiously the NSF has not, to my awareness, done any such study and released it for public consumption.

Well, a group of scientists have recently posted a preprint:

^{Chen, C. Y., Kahanamoku, S. S., Tripati, A., Alegado, R. A., Morris, V. R., Andrade, K., & Hosbey, J. (2022, July 1). Decades of systemic racial disparities in funding rates at the National Science Foundation. OSF Preprints. July 1. doi:10.31219/osf.io/xb57u.}

It reviews National Science Foundation awards (from 1996-2019) and uses demographics provided voluntarily by PIs. They found that the applicant PIs were 66% white, 3% Black, 29% Asian and below 1% for each of American Indian/Alaska Native and Native Hawaiian/Pacific Islander groups. They also found that across the reviewed years, the overall funding rate varied from 22%-34%, so the data were represented as the rate for each group relative to the average for each year. In Figure 1, reproduced below, you can see that applications with white PIs enjoy a nice consistent advantage relative to other groups and the applications with Asian PIs suffer a consistant disadvantage. The applications with Black PIs are more variable year over year but are mostly below average except for 5 years when they are right at the average. The authors note this means that in 2019, there were 798 awards with white PIs above expected value, and 460 fewer than expected awarded with Asian PIs. The size of the disparity differs slightly across the directorates of the NSF (there are seven, broken down by discipline such as Biological Sciences, Engineering, Math and Physical Sciences, Education and Human Resources, etc) but the same dis/advantage based on PI race remains.

Fig 1B from Chen et al. 2022 preprint

It gets worse. It turns out that these numbers include both Research and Non-Research (conference, training, equipment, instrumentation, exploratory) awards. Which represent 82% and 18% of awards, with the latter generally being awarded at 1.4-1.9 times the rate for Research awards in a given year. For white

Fig 3 from Chen et al 2022 preprint FY 13 – 19;
open = Non-Research, closed = Research

PI applications the two types both are funded at higher than the average rate, however significant differences emerge for Black and Asian PIs with Research awards having the lower probability of success.

So why is this the case. Well, the white PI applications get better scores from extramural reviewers. Here, I am not expert in how NSF works. A mewling newbie really. But they solicit peer reviewers which assign merit scores from 1 (Poor) to 5 (Excellent). The preprint shows the distributions of scores for FY15 and FY16 Research applications, by PI race, in Figure 5. Unsurprisingly there is a lot of overlap but the average score for white PI apps is superior to that for either Black or Asian PI apps. Interestingly, average scores are worse for Black PI apps than for Asian PI apps. Interesting because the funding disparity is larger for Asian PIs than for Black PIs. And as you can imagine, there is a relationship between score and chances of being funded but it is variable. Kind of like a Programmatic decision on exception pay or the grey zone function in NIH land. Not sure exactly how this matches up over at NSF but the first author of the preprint put me onto a 2015 FY report on the Merit Review Process that addresses this. Page 74 of the PDF (NSB-AO-206-11) has a Figure 3.2 showing the success rates by average review score and PI race. As anticipated, proposals in the 4.75 (score midpoint) bin are funded at rates of 80% or better. About 60% for the 4.25 bin, 30% for the 3.75 bin and under 10% for the 3.25 bin. Interestingly, the success rates for Black PI applications are higher than for white PI applications at the same score. The Asian PI success rates are closer to the white PI success rates but still a little bit higher, at comparable scores. So clearly something is going on with funding decision making at NSF to partially counter the poorer scores, on average, from the reviewers. The Asian PI proposals do not have as much of this advantage. This explains why the overall success rates for Black PI applications are closer to the average compared with the Asian PI apps, despite worse average scores.

Fig 5 from Chen et al 2022 preprint

One more curious factor popped out of this study. The authors, obviously, had to use only the applications for which a PI had specified their race. This was about 96% in 1999-2000 when they were able to include these data. However it was down to 90% in 2009, 86% in 2016 and then took a sharp plunge in successive years to land at 76% in 2019. The first author indicated on Twitter that this was down to 70% in 2020, the largest one year decrement. This is very curious to me. It seems obvious that PIs are doing whatever they think is going to help them get funded. For the percentage to be this large it simply has to involve large numbers of white PIs and likely Asian PIs as well. It cannot simply be Black PIs worried that racial identification will disadvantage them (a reasonable fear, given the NIH data reported in Ginther et al.) I suspect a certain type of white academic who has convinced himself (it’s usually a he) that white men are discriminated against, that the URM PIs have an easy ride to funding and the best thing for them to do is not to declare themselves white. Also another variation on the theme, the “we shouldn’t see color so I won’t give em color” type. It is hard not to note that the US has been having a more intensive discussion about systemic racial discrimination, starting somewhere around 2014 with the shooting of Michael Brown in Ferguson MO. This amped up in 2020 with the strangulation murder of George Floyd in Minneapolis. Somewhere in here, scientists finally started paying attention to the Ginther Gap. News started getting around. I think all of this is probably causally related to sharp decreases in the self-identification of race on NSF applications. Perhaps not for all the same reasons for every person or demographic. But if it is not an artifact of the grant submission system, this is the most obvious conclusion.

There is a ton of additional analysis in the preprint. Go read it. Study. Think about it.

Additional: Ginther et al. (2011) Race, ethnicity, and NIH research awards. Science, 2011 Aug 19; 333(6045):1015-9. [PubMed]

Research colonialism, analyzing an RFA

I recently fielded a question from a more junior scientist about what, I think, has been termed research colonialism with specificity to the NIH funding disparity known as the Ginther Gap. One of the outcomes of the Hoppe et al 2019 paper, and the following Lauer et al 2021, was a call for a hard look at research on the health issues of communities of color. How successful are grant proposals on those topics, which ICs are funding them, what are the success rates and what are the budget levels appropriated to, e.g. the NIMHD. I am very much at sea trying to answer the question I was asked, which boiled down to “Why is it always majoritarian PIs being funded to do research with communities of color?”. I really don’t know how to answer that or how to begin to address it with NIH funding data that has been generated so far. However, something came across my transom recently that is a place to start.

The NIH issued RFA-MD-21-004 Understanding and Addressing the Impact of Structural Racism and Discrimination on Minority Health and Health Disparities last year and the resulting projects should be on the RePORTER books by now. I was cued into this by a tweet from the Constellation Project which is something doing co-author networks. That may be useful for a related issue, that of collaboration and co-work. For now, I’m curious about what types of PIs have been able to secure funding from this mechanism. According to my RePORTER search for the RFA, there are currently 17 grants funded.

Of the funded grants, there are 4 from NIMHD, 4 from NIDA, 2 from NIA, 1 each from NIMH, NIHNDS, NINR, NICHD, NIGMS, NIDCD, and NCCIH. In the RFA, NIMHD promised 6-7 awards, NIDA 2, NIA 6, NIGMS 4-6 so obviously NIDA overshot their mark, but the rest are slacking. One each was promised for NIMH, NINDS, NICHD, NIDCD and NCCIH, so all of these are on track. Perhaps we will see a few more grants get funded by the time the FY elapses on Sept 30.

So who is getting funded under this RFA? Doing a quick google on the PIs, and admittedly making some huge assumptions based on the available pictures, I come up with

PI/Multi-PI Contact: White woman (2 NIA; 1 NCCIH; 3 NIDA; 1 NIDCD; 1 NIGMS; 1 NINDS); Black woman (1 NIDA; 1 NICHD; 1 NIMHD); Asian woman (1 NIMHD; 1 NIMHD; 1 NINR); White man (1 NIMHD; 1 NIMH)

Multi-PI, non-contact: Asian woman (1 NIA, 1 NIDA, 1 NIMHD); Black woman (2 NIDA, 1 NIMHD); White woman (1 NIDCD; 1 NIGMS; 1 NINR) Black man (1 NIGMS; 1 NIMH); White man (2 NIMH)

I would say the place I am most likely to be off in terms of someone who appears to me to be white but identifies as a person of color would be white women. Maybe 2-3 I am unsure of. I didn’t bother to keep track of how many of the non-contact PIs are on the proposals with white Contact PIs versus the other way around but….I can’t recall seeing even one where a non-contact white PI was on a proposal with a contact PI who is Black or Asian. (There was one award with three white men and one Black man as PIs and, well, does anyone get away with a four PI list that includes no woman anymore?) Anyway… make of that what you will.

I suspect that this RFA outcome is probably slightly better than the usual? And that if you looked at NIH’s studies that deal with communities or color and/or their health concerns more generally it would be even more skewed towards white PIs?

Ginther et al 2011 reported 69.9% of apps in their sample had white PIs, 16.2% had Asian PIs and 1.4% had Black PIs. Hoppe et al 2019 reported (Table S1) 1.5% of applications had Black PIs and 65.7% had white PIs in their original sample. So the 11 out of 17 grants having white PIs/Contact MultiPIs matches expected distribution, as does 3 Asian PIs. Black PIs are over represented since 1-2% of 17 is..zero grants funded. So this was not an opportunity that NIH took to redress the Ginther Gap.

But should it be? What should be the identity of PIs funded to work on issues related to “racism and discrimination” as it applies to “minority health and health disparities”? The “best” as determined by a study section of peer scientists, regardless of applicant characteristics? Regardless of the by now very well established bias against applications with Black PIs?

Someone on twitter asked about the panel that reviewed these grants. You can see from the funded grants on RePORTER that the study section reviewing these proposals was ZMD1 KNL (J1). Do a little web searching and you find that the roster for the 11/15/2021-11/17/2021 meeting is available. A three day meeting. That must have been painful. There are four chairs and a huge roster listed. I’m not going to search out all of them to figure out how many were white on the review panel. I will note that three of the four chairs were white and one was Asian (three of four were MDs, one was a PHD). This is a good place for a reminder that Hoppe et al reported 2.4% of reviewers were Black and 77.8% white in the study sections reviewing proposals for funding in FY2011-2015. I would be surprised if this study section was anything other than majority white.

Research Opportunities for New and “At-Risk” Investigators to Promote Workforce Diversity

NIDA, NIMH, and NINDS have issued a Program Announcement (PAR-22-181) to provide Research Opportunities for New and “At-Risk” Investigators with the intent to Promote Workforce Diversity.

This is issued as a PAR, which is presumably to allow Special Emphasis Panels to be convened. It is not a PAS, however, the announcement includes set-aside funding language familiar to PAS and RFA Funding Opportunity Announcements (FOA).

Funds Available and Anticipated Number of Awards The following NIH components intend to commit the following amounts for the duration of this PAR: NINDS intends to commit up to $10 million per fiscal year, approximately 25 awards, dependent on award amounts; NIDA intends to commit up to $5 million per fiscal year, 12-15 awards, dependent on award amounts; NIMH intends to commit up to $5 million per fiscal year, 12-15 awards, dependent on award amounts; Future year amounts will depend on annual appropriations.

This is a PA typical 3 year FOA which expires June 7, 2025. Reciept dates are one month ahead of standard, i.e., Sept (new R01) / Oct (Resub, Rev, Renew); Jan/Feb; May/Jun for the respective Cycles.

Eligibility is in the standard categories of concern including A) Underrepresented Racial/Ethnic groups, B) Disability, C) economic disadvantage and D) women. Topics of proposal have to be within the usual scope of the participating ICs. Eligibility of PIs is for the familiar New Investigators (“has not competed successfully for substantial, NIH (sic) independent funding from NIH“) and a relatively new “at risk” category.

At risk is defined as “has had prior support as a Principal Investigator on a substantial independent research award and, unless successful in securing a substantial research grant award in the current fiscal year, will have no substantial research grant funding in the following fiscal year.“

So. We have an offset deadline (at least for new proposals), set aside funds, SEPs for review and inclusion of NI (instead of merely ESI) and the potential for the more experienced investigator who is out of funding to get help as well. Pretty good! Thumbs up. Can’t wait to see other ICs jump on board this one.

To answer your first question, no, I have no idea how this differs from the NINDS/NIDA/NIAAA NOSI debacle. As a reminder:

Notice NOT-NS-21-049 Notice of Special Interest (NOSI): NIH Research Project Grant (R01) Applications from Individuals from Diverse Backgrounds, Including Under-Represented Minorities was released on May 3, 2021.

The “debacle” part is that right after NIDA and NIAAA joined NINDS in this NOSI, the Office of the Director put it about that no more ICs could join in and forced a rescinding of the NOSI on October 25, 2021 while claiming that their standard issue statement on diversity accomplished the same goals.

I see nothing in this new PAR that addresses either of the two real reasons that may have prompted the Office of the Director to rescind the original NOSI. The first and most likely reason is NIH’s fear of right wing, anti-affirmative action, pro-white supremacy forces in Congress attacking them. The second reason would be people in high places* in the NIH that are themselves right wing, anti-affirmative action and pro-white supremacy. If anything, the NOSI was much less triggering since it came with no specific plans of action or guarantees of funding. The PAR, with the notification of intended awards, is much more specific and would seemingly be even more offensive to right wingers.

I do have two concerns with this approach, as much as I like the idea.

First, URM-only opportunities have a tendency to put minority applicants in competition with each other. Conceptually, suppose there is an excellent URM qualified proposal that gets really high priority scores from study section and presume it would have also done so in an open, representation-blind study section. This one now displaces another URM proposal in the special call and *fails to displace* a lesser proposal from (statistically probable) a majoritarian PI. That’s less good than fixing the bias in the first place so that all open competitions are actually open and fair. I mentioned this before:

These special FOA have the tendency to put all the URM in competition with each other. This is true whether they would be competitive against the biased review of the regular FOA or, more subtly, whether they would be competitive for funding in a regular FOA review that had been made bias-free(r). […] The extreme example here is the highly competitive K99 application from a URM postdoc. If it goes in to the regular competition, it is so good that it wins an award and displaces, statistically, a less-meritorious one that happens to have a white PI. If it goes in to the MOSAIC competition, it also gets selected, but in this case by displacing a less-meritorious one that happens to have a URM PI. Guaranteed.

The second concern is one I’ve also described before.

In a news piece by Jocelyn Kaiser, the prior NIH Director Elias Zerhouni was quoted saying that study sections responded to his 2006/2007 ESI push by “punishing the young investigators with bad scores”. As I have tried to explain numerous times, phrasing this as a matter of malign intent on the part of study section members is a mistake. While it may be true that many reviewers opposed the idea that ESI applicants should get special breaks, adjusting scores to keep the ESI application at the same chances as before Zerhouni’s policies took effect is just a special case of a more general phenomenon.

So, while this PAR is a great tactical act, we must be very vigilant for the strategic, long term concerns. It seems to me very unlikely that there will be enthusiasm for enshrining this approach for decades (forever?) like the ESI breaks on merit scores/percentiles/paylines. And this approach means it will not be applied by default to all qualifying applications, as is the case for ESI.

Then we get to the Oppression Olympics, an unfortunate pitting of the crabs in the barrel against each other. The A-D categories of under-representation and diversity span quite a range of PIs. People in each category, or those who are concerned about specific categories, are going to have different views on who should be prioritized. As you are well aware, Dear Reader, my primary concern is with the Ginther gap. As you are aware, the “antis” and some pro-diversity types are very concerned to establish that a specific person who identifies as African-American has been discriminated against and is vewwwwy angweee to see any help being extended to anyone of apparent socio-economic privileges who just so happens to be Black. Such as the Obama daughters. None of us are clean on this. Take Category C. I have relatively recently realized that I qualify under Category C since I tick three of the elements, only two are required. I do not think that there is any possible way that my qualification on these three items affects my grant success in the least. To do so would require a lot of supposing and handwaving. I don’t personally think that anyone like me who qualifies technically under Category C really should be prioritized against, say, the demonstrated issue with the Ginther gap. These are but examples of the sort of “who is most disadvantaged and therefore most deserving” disagreement that I think may be a problem for this approach.

Why? Because reviewers will know that this is the FOA they are reviewing under. Opinions on the relative representation of categories A-D, Oppression Olympics and the pernicious stanning of “two-fers” will be front and present. Probably explicit in some reviews. And I think this is a problem in the broader goals of improving equity of opportunity and in playing for robust retention of individuals in the NIH funded research game.

__

*This is going to have really ugly implications for the prior head of the NIH, Francis Collins, if the PAR is not rescinded from the top and the only obvious difference here is his departure from NIH.

Really, Dartmouth College Psych Dept? Really???

The Department of Psychological and Brain Sciences at Dartmouth College (yes, that department) has announced a most unusual academic position that they are seeking to fill.

The @Dartmouth Department of Psychological and Brain Sciences invites applications for a Faculty Fellow: a postdoc appointment that will *automatically* convert to a regular full-time tenure track assistant professorship after 2 years! Details: https://t.co/uYOFSND25s

— Dartmouth Psychological and Brain Sciences (@DartmouthPBS) September 21, 2021

Huh? Well, let us click through and read the details.

The Department of Psychological and Brain Sciences at Dartmouth College invites applications for a Faculty Fellow, a two-year residential postdoctoral appointment, that will convert automatically to a regular full-time tenure-track appointment as Assistant Professor. Faculty Fellows are part of a cohort of faculty committed to increasing diversity in their disciplines. We are interested in applicants whose research can connect to and/or bridge between any foci in our department including behavioral, cognitive, social and affective psychology and neuroscience. We are especially interested in candidates who have a demonstrated ability to contribute to Dartmouth’s diversity initiatives in STEM research, such as the Women in Science Program, E. E. Just STEM Scholars Program, and Academic Summer Undergraduate Research Experience (ASURE).

That’s about it. The rest is boilerplate with minimal details, including about the salary and resources being offered. So we have to assume “postdoctoral appointment” means the usual for a postdoc. Salary something along the lines of the NRSA scale and no individual resources such as a nice startup package or research space. Maybe there will be, but it is not on evidence in the job solicitation.

This is CLEARLY a DEI hire. An attempt to diversity a faculty that looks to my eye like it could use some diversification.

Instead of just hiring a person at the faculty level directly, they will be getting faculty level effort and behavior out of someone for the low, low price of a postdoc stipend. With a guarantee of “automatic” conversion which one, frankly, doubts will be iron clad.

This is so dismally emblematic of the institutional efforts to respond to the pressure to diversify their faculty.

Are any of you seeing similar proposals lauched at your University? What is the rationale here? What is the justification for this over just creating new faculty lines and hiring into them?

I can think of a couple of rationalescuses.

It is some sort of tenure clock manipulation. If they think, for whatever reason, that someone that will be able to contribute to “Dartmouth’s diversity initiatives in STEM research” will have a hard time making tenure on the usual schedule in their Department, this could be the reason for the plan. This would be an extra red flag warning to any applicant, of course. If the Department can’t get behind valuing these contributions as a substitute for their other expectations, and only see them as add-on effort that delays “real progress”, then this person will always be at odds with an unsupportive Department. Tenure is a risky proposition, no matter how long the decision is delayed.

It could be some sort of “we can get this approved quickly but oh how hard it is to get a new tenure line approved for this cycle” thing. Yeah, well that questions the commitment of the College and the “automatic” conversion. So surely this isn’t going to be raised.

A colleague from elsewhere indicated that something like this is being tried in their Department. The rationale is, from what I can tell, that scientist of color are reluctant to take on postdoctoral training (pretty sure I’ve seen data on that mentioned somewhere) and that this leak in the pipeline could be addressed by offering faculty positions earlier. Ok, I definitely buy that more security of a career would be helpful to keep promising younger scientists from bailing on the academic track before or during the expected postdoctoral interval. But. But, but but. Why not just hire straight into a faculty position? Course relief, service relief, etc, is already standard operating procedure. If a Department or University (or College) thinks this needs to be extended two or three years longer for these earlier-career hires, so be it.

This brings us to the longer arc of wage manipulation in the individual sense and in the industry sense.

If these Departments who are all really concerned about DEI and are launching various hiring initiatives were serious, they would have to be out there competing with each other for the existing pool of academic scientists in more or less the same position as their usual hires. As we know, there aren’t a lot of them, particularly when it comes to African-American scientists and some other key Federally defined underrepresented groups. So, according to market forces the Departments would have to PAY. More salary. More support. More startup cash. More housing / relocation allowances. More spousal hire opportunity. More everything.

This plan short circuits that by locking in candidates before they are as competitive on the open market. When they are still relatively desperate and/or think this is a great opportunity to jump ahead on the career arc. And as more Departments catch wind of this excellent strategy they are more likely to opt for this can-kicking strategy and less likely to PAY to get those who are currently trained to the usual point of faculty hires.

Making sense of the T32 effect on the NIH funding disparity

A new blog post from Mike Lauer, Deputy Director for Extramural Research at the NIH, presents new data on grant award (K and R01equivalent) to applicants assuming they had a prior appointment on a NIH funded T32 training grant. This particular post included some demographic analysis which addressed URM vresus majoritarian status. Leaving aside for a moment the obvious concerns about who gets selected for T32 appointments and where and with whom they are correspondingly training, I was curious about the relative advantage.

Now, the first thing you will recognize is that the critical data are collapsed across all URM groups, although this post doesn’t seem to specify if Asian is included in non-URM, the total N in Table 1 and Table 3 suggest this is the case. This is a little disappointing of course, since the original Ginther report found such striking differences in award rates across URM groups. There was an Asian PI disparity and a Hispanic PI lackthereof, so would it not be super important to keep these groups separate for this post-T32 analysis? Of course it would.

But what got me scratching my head was the presentation of the percentages in Table 3. The show, for example, that 12.9% of non-URM T32 trainees eventually became PI on a funded R01 whereas only 9.8% of URM trainees became PI on a funded R01. That’s all well and good but it obscures the success rate because it doesn’t account for the differential submission rate (24.5% of nonURM T32 trainees eventually submitted an R01, 21.8% of URM trainees).

Doing a little math here, I make it out to be a 52.9% award rate for non-URM applicants and a 45.1% rate for URM applicants. This is a 7.8 percentage point differential which means the URM applicants have a success rate that is 85.3% of the non-URM applicant success rate.

Now we can put this in familiar terms. The Hoppe report found that Black applicants had a success rate that was 60.5% of that enjoyed by white applicants (it was 60.4 in Ginther et al, 2011). So if we take this 24.8 percentage point differential and divide it by the 39.5 percentage point deficit for Black applicants in the Hoppe sample…we end up with about 63% of the Black/white gap reported in Hoppe. The combination of T32 training history, combining of all URM together and adding Asian PIs to the non-URM group closes 63% of the gap.

See my problem here? We want all this broken down so we can answer the most simple question, does T32 training close the gap between Black and white PIs and if so to what extent.

THEN we can go on to ask about other URM populations and the effect of adding the Asian* T32 participants to the non-URM pile.

*reminder that Ginther found that applications with Asian PIs with domestic doctorates had success rates similar to those with white PIs. There is a major complication here with T32 eligibility for Asian-Americans versus Asians who were not T32 eligible as postdocs.

NOSI for URM Investigators: Choose Wisely

I was recently describing Notice NOT-NS-21-049 Notice of Special Interest (NOSI): NIH Research Project Grant (R01) Applications from Individuals from Diverse Backgrounds, Including Under-Represented Minorities in the context of the prior NIH Director’s comments that ESI scores got worse after the news got around about relaxed paylines.

One thing that I had not originally appreciated was the fact that you are only allowed to put one NOSI in Box 4b.

Which means that you have to choose. If you qualify as an individual from diverse backgrounds you could use this, sure. But that means you cannot use a NOSI that is specific to the topic you are proposing.

This is the usual NIH blunder of stepping on their own junk. How many ways can I count?

Look, the benefit of NOSI (and the predecessor, the Program Announcement) is uncertain. It seemingly only comes into play when some element of Program wishes to fund an award out of the order of review. Wait, you say, can’t they just do that anyway for whatever priority appears in the NOSI? Yes, yes they can….when it comes to the topic of the grant. So why do NOSI exist at all?

Well…one presumes it is because elements of Program do not always agree on what should be funded out of order of review. And one presumes there is some sort of conflict resolution process. During which the argument that one grant is related to the Programmatic Interest formally expressed in the NOSI has some weight or currency. Prioritizing that grant’s selection for funding over the identically-percentiled grant that does not mention a NOSI.

One still might wonder about a topic that fits the NOSI but doesn’t mention the NOSI directly. Well, the threat language at the bottom of some of those NOSI, such as oh I don’t know this one, is pretty clear to me.

• For funding consideration, applicants must include “NOT-DA-21-006” (without quotation marks) in the Agency Routing Identifier field (box 4B) of the SF424 R&R form. Applications without this information in box 4B will not be considered for this initiative.

Applications nonresponsive to terms of this NOSI will not be considered for the NOSI initiative.

So what is a PI to do? Presumably the NOSI has some non-negligible value and everyone is motivated to use those if possible. Maybe it will be the difference between a grey zone pickup and not, right? If your ideas for this particular grant proposal fit with something that your favorite IC has gone to the trouble if mentioning in a NOSI…well….dang it….you want to get noticed for that!

So what can you do if you are a person underrepresented who qualifies for the NOSI NOT-NS-21-049 ? The value of this one is uncertain. The value of any other NOSI for your particular application is likewise uncertain. We know perfectly well the NIH as a whole is running scared of right wing political forces when it comes to picking up grants. We know that this NOSI may be related to the well meaning ICs’ staff having difficulty getting PI demographic information and could simply be a data collection strategy for them.

Cynical you say? Well I had a few exchanges with a fairly high up Program person who suggested to me that perhaps the strategy was to sneak the “extra” NOSI into the Abstract of the proposal. This would somehow get it in front of Program eyes. But….but….there’s the “will not be considered” boilerplate. Right? What does this mean? It is absolutely maddening for PIs who might like to take advantage of this new NOSI which one might think would be used to fix the Ginther Gap. It is generally enraging for anyone who wants to see the Ginther Gap addressed.

It makes me positively incandescent to contemplate the possibility that the mere announcing of this NOSI will lead to study sections giving even worse scores to those applications, without any real assistance coming from Program.

A couple of more thoughts. This doesn’t apply to anything other than an R01 application, which is nuts. Why not apply it to all investigator initiated mechanisms? Trust me, underrepresented folks would like a leg up on R21 and R03 apps as well. These very likely help with later R01 getting, on a NIH wide statistical basis. You know, the basis of the Ginther Gap. So why not include other mechs?

And did you notice that no other ICs have joined? NINDS issued the NOT on May 3 and they were rapidly joined by NIDA (May 6) and NIAAA (May 11). All in good time for the June 5 and July 5 submission rounds. Since then….crickets. No other ICs have joined in. Weird, right?

I was on a Zoom thing awhile back where a highly authoritative Program person claimed that the Office of the Director (read: Francis Collins) had put a hold on any more ICs joining the NINDS NOSI.

Why? Allegedly because there was a plan to make this more general, more NIH wide all at one fell swoop.

Why? Who the heck knows. To cover up the reluctance of some of the ICs that would not be joining the NOSI if left up to their own devices? If so, this is HORRENDOUS, especially give the above mentioned considerations for choosing only one NOSI for Box 4b. Right? If they do extend this across all NIH, how would any PI know that their particular IC has no intention whatsoever of using this NOSI to fund some grants? So maybe they choose to use it, for no help, while bypassing another NOSI that might have been of use to them.

NINDS Issues NOSI Requesting Applications from URM Investigators

Notice NOT-NS-21-049 Notice of Special Interest (NOSI): NIH Research Project Grant (R01) Applications from Individuals from Diverse Backgrounds, Including Under-Represented Minorities was released on May 3, 2021.

The NOSI is the new Program Announcement, for those who haven’t been keeping track. As with the old PA the direct benefit is not obvious. There is no set aside funding or promise to fund any applications at all. In the context of Ginther et al 2011, Hoppe et al 2019, the discussions of 2020 and the overall change in tone from the NIH on diversity matters, it is pretty obvious that this is designed merely to be the excuse. This PA, sorry NOSI, is what will permit Program to pick up grants on the (partial?) basis of the PI’s identity.

What identities? Well the NOSI re-states the categories A, B and C that are familiar from other similar documents.

A. Individuals from racial and ethnic groups that have been shown by the National Science Foundation to be underrepresented in health-related sciences on a national basis…

B. Individuals with disabilities, who are defined as those with a physical or mental impairment that substantially limits one or more major life activities

C. Individuals from disadvantaged backgrounds, defined as those who meet two or more of the following criteria…

And then there is a statement about gender intersectionality to close out.

GREAT! Right?

Yeah, it is. To the extent this is used to figure out a way to start working the naked, quota based, top down, heavy handed affirmative action that has been benefiting ESI applicants since 2007 on the NIH funding disparity identified in Ginther and Hoppe, this is a win. From the very start of hearing about Ginther I‘ve been talking about exception pay rapid solutions and this only heated up with the disingenuous claim that exception pay was not accelerating the disparity which was made in Hoppe et al. The NOSI allows pickups/exception pay, for sure, under the “special interest” idea. I don’t know if this will end up generating explicit payline benefits on a universal categorical basis as has been done at many ICs for ESI applications. The difference, of course, is that the former is much more variable and subject to biases that are expressed by Program Officers individually and collectively. Explicit rules would be better…..ish. It’s complicated.

In a news piece by Jocelyn Kaiser, the prior NIH Director Elias Zerhouni was quoted saying that study sections responded to his 2006/2007 ESI push by “punishing the young investigators with bad scores”. As I have tried to explain numerous times, phrasing this as a matter of malign intent on the part of study section members is a mistake. While it may be true that many reviewers opposed the idea that ESI applicants should get special breaks, adjusting scores to keep the ESI application at the same chances as before Zerhouni’s policies took effect is just a special case of a more general phenomenon.

NIH grant reviewers have a pronounced tendency to review grant proposals with an eye to “fund it” versus “don’t fund it”. Continual exhortations from SROs that panels do not make funding decisions, that they should review merit on a mostly continuous scale and that they should spread scores has minimal impact on this. Reviewers have a general idea of what scores will result in funding and what will not and they score accordingly*. I have mentioned that when I first started on study section the SRO would actually send us score distributions as part of the effort to get us to spread scores. INVEVITABLY the scores would stack up around the perceived funding line. Across a couple of years one could even see this move in tandem (with a round or two lag, obv) with what was funding at the 2-3 ICs that most grants were assigned to.

One interpretation of the “punishing” phenomenon is simply that panels were doing what they always do (as I assert anyway) in matching scoring to their perception of the payline and their gut feeling about whether a given app was deserving. What this assumes, of course, is that whatever biases were keeping the applications of the ESI-qualifying individuals from getting good scores in the past were still present and the reviewers were simply continuing the same behavior.

My concern with the new NOSI is, of course, that something similar will happen with the applications that qualify for this NOSI. There is the potential for a growing general suspicion (assumption? bias?) among study section reviewers that “oh, those URM PIs get special breaks” and then the scores will get even worse than they were before this idea started to percolate around the culture. It might be accelerated if the ICs generate explicit statements of relaxed paylines…but the campfire chatter about how the NOSI is being used will be sufficient.

Vigilance!

Vigilance is the thing. NIH cannot be permitted to put this in place, pay no attention to the results and “suddenly realize” five or ten years later that things are not working according to design.

__

*generally. Reviewers can be mistaken about paylines. They can be miscalibrated about scores and percentiles. They have a limited picture which only reflects their own knowledge of what is being funded. But still, there is an aggregate effect.

A lesson for DEI strategies from the NIH ESI policy

The Director of the NIH, in the wake of a presentation to the Advisory Committee to the Director meeting, has issued a statement of NIH’s commitment to dismantle structural racism.

Toward that end, NIH has launched an effort to end structural racism and racial inequities in biomedical research through a new initiative called UNITE, which has already begun to identify short-term and long-term actions. The UNITE initiative’s efforts are being informed by five committees with experts across all 27 NIH institutes and centers who are passionate about racial diversity, equity, and inclusion. NIH also is seeking advice and guidance from outside of the agency through the Advisory Committee to the Director (ACD), informed by the ACD Working Group on Diversity, and through a Request for Information (RFI) issued today seeking input from the public and stakeholder organizations. The RFI is open through April 9, 2021, and responses to the RFI will be made publicly available. You can learn more about NIH’s efforts, actions, policies, and procedures via a newly launched NIH webpage on Ending Structural Racism aimed at increasing our transparency on this important issue.

This is very much welcome, coming along as it does, a decade after Ginther and colleagues showed that Black PIs faced a huge disadvantage in getting their NIH grants funded. R01 applications with Black PIs were funded at only 58% of the rate that applications with white PIs were funded.

Many people in the intervening years, accelerated after the publication of Hoppe et al 2019 and even further in the wake of the murder of George Floyd at the hands of the Minneapolis police in 2020, have wondered why the NIH does not simply adopt the same solution they applied to the ESI problem. In 2006/2007 the then-Director of NIH, Elias Zerhouni, dictated that the NIH would practice affirmative action to fund the grants of Early Stage Investigators. As detailed in Science by Jocelyn Kaiser

Instead of relying solely on peer review to apportion grants, [Zerhouni] set a floor—a numerical quota—for the number of awards made to new investigators in 2007 and 2008.

A quota. The Big Bad word of anti-equity warriors since forever. Gawd forbid we should use quotas. And in case that wasn’t clear enough

The notice states that NIH “intends to support new investigators at success rates comparable to those for established investigators submitting new applications.” In 2009, that will mean at least 1650 awards to new investigators for R01s, NIH’s most common research grant.

As we saw from Hoppe et al, the NIH funded 256 R01s with Black PIs in the interval from 2011-2015, or 51 per year. In a prior blog post I detailed how some 119 awards to poorer-scoring applications with white PIs could have been devoted to better-scoring proposals with Black PIs. I also mentioned how doing so would have moved the success rate for applications with Black PIs fro 10.7% to 15.6% whereas the white PI success rate would decline from 17.7% to 17.56%. Even funding every single discussed application with a Black PI (44% of the submissions) by subtracting those 1057 applications from the pool awarded with white PIs would reduce the latter applications’ hit rate only to 16.7% which is still a 56% higher rate than the 10.7% rate that the applications with Black PIs actually experienced.

I have been, and continue to be, an advocate for stop-gap measures that immediately redress the funding rate disparity by mandating at least equivalent success rates, just as Zerhouni mandated for ESI proposals. But we need to draw a key lesson from that episode. As detailed in the Science piece

Some program directors grumbled at first, NIH officials say, but came on board when NIH noticed a change in behavior by peer reviewers. Told about the quotas, study sections began “punishing the young investigators with bad scores,” says Zerhouni. That is, a previous slight gap in review scores for new grant applications from first-time and seasoned investigators widened in 2007 and 2008, [then NIGMS Director Jeremy] Berg says. It revealed a bias against new investigators, Zerhouni says.

I don’t know for sure that this continued, but the FY 2012 funding data published by Kienholz and Berg certainly suggest that several NIH ICs continued to fund ESI applications at much lower priority scores/percentiles than were generally required for non-ESI applications to receive awards. And if you examine those NIH ICs pages that publish their funding strategy each year [see the writedit blog for current policy links], you will see that they continue to use a lower payline for ESIs. So. From 2007 to 2021 that is a long interval of policy which is not “affirmative action”, just, in the words of Zerhouni, “leveling the playing field”.

The important point here is that the NIH has never done anything to get to the “real reason” for the fact that early stage investigators proposals were being scored by peer review at lower priority than they, NIH, desired. They have not undergone spasms of reviewer “implicit bias” training. They have not masked the identity of the PI or done anything suggesting they think they can “fix” the review outcome for ESI PIs.

They have accepted the fact that they just need to counter the bias.

NIH will likewise need to accept that they will need to fund Black PI applications with a different payline for a very long time. They need to accept that study sections will “punish*” those applications with even worse scores. They will even punish those applications with higher ND rates. And to some extent, merely by talking about it this horse has left the stall and cannot be easily recalled. We exist in a world where, despite all evidence, white men regularly assert with great confidence that women or minority individuals have all the advantages in hiring and career success.

So all of this entirely predictable behavior needs to be accounted for, expected and tolerated.

__

*I don’t happen to see it as “punishing” ESI apps even further than whatever the base rate is. I think reviewers are very sensitive to perceived funding lines and to reviewing grants from a sort of binary “fund/don’t fund” mindset. Broadcasting a relaxed payline for ESI applications almost inevitably led to reviewers moving their perceived payline for those applications.

On targeting NIH funding opportunities to URMs: The Lauer edition

I have long standing doubts about certain aspects of funding mechanisms that are targeted to underrepresented individuals. This almost always has come up in the past in the context of graduate or postdoctoral fellowships and when there is a FOA open to all, and a related or parallel FOA that is directed explicitly at underrepresented individuals. For example see NINDS F31, K99/R00 , NIGMS K99/R00 initiatives, and there is actually a NIH parent F32 – diversity as well).

At first blush, this looks awesome! Targeted opportunity, presumably grant panel review that gives some minimal attention to the merits of the FOA and, again presumably, some Program traction to fund at least a few.

My Grinchy old heart is, however, suspicious about the real opportunity here. Perhaps more importantly, I am concerned about the real opportunity versus the opportunity that might be provided by eliminating any disparity of review that exists for the review of applications that come in via the un-targeted FOA. No matter the FOA, the review of NIH grants is competitive and zero sum. Sure, pools of money can be shifted from one program to another (say from the regular F31 to the F31-diversity) but it is rarely the case there is any new money coming in. Arguing about the degree to which funding is targeted by decision of Congress, of the NIH Director, of IC Directors or any associated Advisory Councils is a distraction. Sure NIGMS gets a PR hit from announcing and funding some MOSAIC K99/R00 awards…but they could just use those moneys to fund the apps coming in through their existing call that happen to have PIs who are underrepresented in science.

The extreme example here is the highly competitive K99 application from a URM postdoc. If it goes in to the regular competition, it is so good that it wins an award and displaces, statistically, a less-meritorious one that happens to have a white PI. If it goes in to the MOSAIC competition, it also gets selected, but in this case by displacing a less-meritorious one that happens to have a URM PI. Guaranteed.

These special FOA have the tendency to put all the URM in competition with each other. This is true whether they would be competitive against the biased review of the regular FOA or, more subtly, whether they would be competitive for funding in a regular FOA review that had been made bias-free(r).

I was listening to a presentation from Professor Nick Gilpin today on his thoughts on the whole Ginther/Hoppe situation (see his Feature at eLife with Mike Taffe) and was struck by comments on the Lauer pre-print. Mike Lauer, head of NIH’s office of extramural awards, blogged and pre-printed an analysis of how the success rates at various NIH ICs may influence the funding rate for AA/B PIs. It will not surprise you that this was yet another attempt to suggest it was AA/B PIs’ fault that they suffer a funding disparity. For the sample of grants reviewed by Lauer (from the Hoppe sample), 2% were submitted with AA/B PIs, NIH-wide. The percentage submitted to the 19 individual funding ICs he covered ranged from 0.73% to 14.7%. This latter institute was the National Institute on Minority Health and Health Disparities (NIMHD). Other notable ICs of disproportionate relevance to the grants submitted with AA/B PIs include NINR (4.6% AA/B applications) and NICHD (3%).

So what struck me, as I listened to Nick’s take on these data, is that this is the IC assignment version of the targeted FOA. It puts applications with AA/B investigators in higher competition with each other. “Yeahbutt”, you say. It is not comparable. Because there is no open competition version of the IC assignment.

Oh no? Of course there is, particularly when it comes to NIMHD. Because these grants will very often look like a grant right down the center of those of interest to the larger, topic-focused ICs….save that it is relevant to a population considered to be minority or suffering a health disparity. Seriously, go to RePORTER and look at new NIMHD R01s. Or heck, NIMHD is small enough you can look at the out year NIMHD R01s without breaking your brain since NIHMH only gets about 0.8% of the NIH budget allocation. With a judicious eye to topics, some related searches across ICs, and some clicking on the PI names to see what else they may have as funded grants, you can quickly convince yourself that plenty of NIMHD awards could easily be funded by a related I or C with their much larger budgets*. Perhaps the contrary is also true, grants funded by the parent / topic IC which you might also argue would fit at NIMHD, but I bet the relative percentage goes the first way.

If I am right in my suspicions, the existence of NIMHD does not necessarily put more aggregate money into health disparities research. That is, more than that which could just as easily come out of the “regular” budget. The existence of NIMHD means that the parent IC can shrug off their responsibility for minority health issues or disparity issues within their primary domains of drug abuse, cancer, mental health, alcoholism or what have you. Which means they are likewise shrugging off the AA/B investigators who are disproportionately submitting applications with those NIMHD-relevant topics and being put in sharp competition with each other. Competition not just within a health domain, but across all health domains covered by the NIH.

It just seems to me that putting the applications with Black PIs preferentially in competition with themselves, as opposed to making it a fair competition for the entire pool of money allocated to the purpose, is sub optimal.

__

*Check out the descriptions for MD010362 and CA224537 for some idea of what I mean. The entire NIMHD budget is 5% as large as the NCI budget. Why, you might ask, is NCI not picking up this one as well?

Merit of NIH Grants Depends on the Reviewer’s Opinion

This is not news to this audience but it bears addressing in as many ways as possible, in the context of the Hoppe et al 2019 and Ginther et al 2011 findings. Behind most of the resistance to doing anything about the funding disparity for investigators and, as we’re now finding out, topics is still some lingering idea that the NIH grant selection process is mostly about merit.

Objective merit. Sure we sort of nod that we understand that there is some wiggle room but overall it is difficult to find anyone that appears to understand something in a deep way.

“Merit” of NIH grants is untethered to anything objective. It relies on the opinion of the peer reviewers. The ~3 reviewers who are assigned to do deep review and the members of the panel (which can be 20-30ish folks) who vote scores after the discussion.

For NIH grant review-experienced tweeps, do you ever experience a panel voting a fundable score for an application you personally think does not deserve funding?

— Drug Monkey (@drugmonkeyblog) January 27, 2021

This particular dumb twitter poll shows that 77% of experienced reviewers either occasionally or regularly have the experience of thinking a grant that should not receive funding is very likely to do so.

For NIH grant review-experienced tweeps, do you ever experience a panel voting a clearly not-fundable score for an application you personally think quite obviously deserves funding?

— Drug Monkey (@drugmonkeyblog) January 27, 2021

and this other dumb little twitter poll shows that 88% of experienced NIH grant reviewers either occasionally or frequently experience a panel voting a non-fundable score for a grant they think deserves funding.

It will not escape you that individual reviewers tend to think a lot more grants should be funded than can be funded. And this shows up in the polls to some extent.

This is not high falutin’ science and it is possible we have some joker contamination here from people who are not in fact NIH review experienced.

But with that caveat, it tends to support the idea that the mere chance of which individuals are assigned to review a grant can have a major effect on merit. After all, the post-discussion scores of these individuals tends to significantly constrain the voting. But the voting is important too, since panel members can go outside the range or decide en masse to side with one or the other ends of the post-discussion range.

Swap out the assigned reviewers for a different set of three individuals and the outcomes are likely to be very different. Swap out one panel for another and the tendencies could be totally different. Is your panel heavy in those interested in sex differences and/or folks heavily on board with SABV? Or is it dominated by SABV resisters?

Is the panel super interested in the health effects of cannabis and couldn’t give a fig about methamphetamine? What do YOU think is going to come out of that panel with fundable scores?

Does the panel think any non-mammalian species is horrible for modeling human health and should really never be funded? Does the panel geek away at tractable systems and adore anything fly or worm driven and complain about the lack of manipulability available in a rat?

Of course you know this. These kinds of whines and complaints are endemic to fireside chats whenever two or more NIH grant-seeking investigators are present!

But somehow when it is a disparity of race or of topics of interest to minority communities in the US, such as from Hoppe et al 2019, then nobody is concerned. Even when there are actual data on the table showing a funding disparity. And everyone asks their “yeahbutwhatabout” questions, springing right back into the mindset that at the very core the review and selection of grants is about merit. The fact their worm grant didn’t get selected is clear evidence of a terrible bias in the NIH approach. The fact African-American PIs face a payline far lower than they do…..snore.

Because in that case it is about objective merit.

And not about the coincidence of whomever the SRO has decided should review that grant.

Happy 2021!!!!!

As we turn our backs on 2020, a real jackass of a year, I wish you all good fortune. May your grant applications be funded, your papers be published and your exciting new science keep you jumping to see new data every week.

I’m not a big fan of huge sweeping goals and resolutions, personally. And right now, I am going to be satisfied with putting my research program back together.

As you will recall, I changed jobs in early 2019. We were juuuuusst getting the laboratory into something resembling operational shape in January 2020. And then Covid hit.

This has been very much not-fun for me. As you can imagine. The hardest thing of all is the loss of the data stream. As I may have mentioned before this is the bulk of the reason why I do this job. To see the data.

But I also picked up some new responsibilities in 2020, in part due to the job change and in part due to the death of George Floyd, murdered by Minneapolis police officers in May. Our various academic institutions had a bit of a moment. This caused an opening. A “strike while the iron is hot” moment. From my perspective anyway. So, despite a certain weariness with institutional efforts on diversity, I’m back in the fight. I say yes to way more things than I would have prior to May of 2020. I am accepting more obvious tokenism offers. I am bearing down.

I plan to continue that for a little bit more. Until the steam seems to have escaped and the ingot as cold as the earth.

I have been delighted to see many of you doing the same. Recognizing this moment and getting down to work with hammer and tongs. I know you can’t all sustain these efforts forever. Do it as long as you can. And then rest knowing you did your part.

Happy New Year.

Black PIs working on the “right” topics are at a further NIH funding disadvantage

One of the potential takeaway messages from the Hoppe et al 2019 finding, and the Open Mike blogpost, is that if Black PIs want to have a better success rate for their applications, perhaps they should work on the right topics. The “right” topics, meaning the ones that enjoy the highest success rates. After all, the messaging around the release of Hoppe was more or less this: Black PI apps are only discriminated against because they are disproportionately proposed on topics that are discriminated against.

(Never mind that right in the Abstract of Hoppe they admit this only explains some 20% of the funding gap.)

We find, however, a curious counter to this message buried in the Supplement. I mentioned this in a prior post but it bears posting the data for a more memorable impact.

The left side of Figure S6 in Hoppe et al. shows the percent of applications within both the African-American/Black and WHite distributions that were submitted which landed in topic clusters across the success rate quintiles. The 1st is the best, i.e, the most readily funded topic cluster. We can see from this that while applications with white PIs are more or less evenly distributed, the applications with Black PIs are less frequently landing in the best funded topic clusters and more frequently landing in the lowest funded topic clusters. Ok, fine, this is the distributional description that underlies much of the takeaway messaging. On the right side of Figure S6, there is a Table. No idea why they chose that instead of a graph but it has the tendency to obscure a critical point.

Here I have graphed the data, which is the success rate for applications which fall into the topic-success quintiles by the race of the PI. This, first of all, emphasizes the subvocalized admission that even in the least-fundable topic clusters, applications with white PIs enjoyed a success advantage. Actually this main effect was present in each quintile, unsurprisingly. What also immediately pops out, in a way it does not with the table representation, is that in the best funded topic area the advantage of applications with white PIs is the greatest. Another way to represent this is by calculating the degree to which applications with Black PIs are disadvantaged within each quintile.

This represents the success for applications with Black PIs as a percentage of the success for applications with white PIs. As you can see, the biggest hit is at the first and fifth quintiles with the applications faring the best at the middle topic-success quintile. Why? Well one could imagine all kinds of factors having to do with the review of applications in those topic domains. The OpenMike blog post on ICs with lower funding rates (because they have tiny budgets, in part) may explain the fifth quintile but it wouldn’t apply to the top quintile. In fact quite the contrary. Ok, this depiction speaks to the relative hit to success rates within quintile. But the applicant might be a little more interested in the raw percentile hit, especially given the cumulative probability distributions we were discussing yesterday. Recall, the average difference was 7 percentile points (17.7% for Wh PI apps vs 10.7% for Black PI apps).

The disparity is highest in the first quintile. It is a hit of 10.8 percent, as opposed to the all-apps average hit of 7.0 percent.

Obviously we cannot draw much more from the available data. But it certainly cautions us that pushing Black applicants to work on the “right” topics is not a clear solution and may even be counter productive. This is on the acute level of a current PI deciding what to propose in an application, and what to pursue with a multi-application strategy over time. But it also, importantly, serves up a caution for pipeline solutions that try to get more Black trainees into the right labs so that they will work on the right topics using, in Dr. Collins’ parlance, “upgraded” “methodologies”. If this topic/race disparity is not resolved by the time these new trainees hit Assistant Professor stage, we are going to push more Black Professors into research domains that are even harder to succeed in.

We may eventually get some more insight. CSR promised this summer to start looking into study section behavior more closely. It may be that fewer applications from Black PIs in the most successful topic domains is due to disproportionately fewer Black PIs in those fields which leads to fewer of them on the relevant study sections. Even absent that factor, a lower presence in the fields of interest may drive more implicit or explicit bias against the ones that do choose those fields. We just can’t tell without more information about the constitution of study sections and the success rates that emerge from them. Oh, and the exception pay behavior of the Branches and Divisions within each IC. That’s also important to examine as it may relate to topic domain.

The cumulative probability of getting NIH grants

One of the career strategies we have discussed numerous times in various contexts is how many grant applications one should be submitting to the NIH. I have been a consistent advocate for …more. This is a recognition that success rates on a per-application basis have been below 20% for most of my career. Obviously this particular target number varies a lot. Sometimes we are talking about paylines, since that seems to be a hard target for success. Recent paylines from the NCI have been in the high single digits- 7-9%. Or we may talk about NIH-wide success rates overall, accounting for not just payline funding but pickups above the payline. These numbers change from year to year but mid to upper teens is a pretty fair general estimate.

My usual argument is that investigators who want to get funded should start with the assumption that they are no better than anyone else and need to base their strategy off the average success rate….at the very least.

Dumb old me, math challenged in the extreme, may have even expressed this as something like “If the payline is 10%, you need to put in 10 applications to even be in the game”. The more math savvy of you immediately chime in to correct me about calculating cumulative probabilities. This is not a difficult concept. I get it. But my mind likes to forget about it and I’ve never taken a blog whack at this issue directly, that I can recall.

Thanks to this handy binomial probability tool I googled up, we can now contemplate the abyss. Let us suppose a per-application success rate of 17.7%. Mid to upper teens, a decent place to start our discussion. And let us gate on the cumulative probability of at least one award. Well, if you put in 5 applications, your odds of one of them funding is 62.2% and if you put in 10 applications, this is 85.7%. Not too shabby. But it puts a very fine point that probabilities of award do not add. Fifteen applications are required to get to a 95% cumulative probability of at least one being awarded.

Reminder: We are not talking payline here. We are talking the 17.7% hit rate for the NIH wide average of everything that goes into awards including all of those various pickup behaviors. If you want the assurance of making the payline at a place like NCI, well…..Lord help you, amirite? That sounds totally rough and brutal and even unfair.

Now. Suppose that for some reason, say that your skin reflectance categorizes you as Black instead of white, your success rate was 10.7% instead of 17.7%.

Your chances are, of course, somewhat different. The cumulative advantage of putting in more grants, aka working harder, accrues less surely here as well. I’ve color coded a few ~equivalent cumulative probabilities for convenience. Using the Hoppe success rates, a Black applicant has to put in 9 proposals to get approximately the same 62% chance a white applicant achieves with only 5 applications. This same PI would have to put in 18 proposals to approximate the 86% hit rate the white PI gets with only 10 applications. About 25 proposals to get the 95% hit rate enjoyed by white applicants who put in 15 proposals.

Insert my favorite staring eyes emojii available on other platforms.

I would estimate that many Black folks, in the academy and elsewhere, are somewhat used to the idea that they need to grind a bit harder to achieve. At some level it probably doesn’t even really bother some on the day to day.

But this is a LOT harder. Putting in just one grant proposal is not easy. Particularly when you are a brand new Professor. But it is not trivial, even when you are a Full professor with some time and experience under your belt. [ Oh, btw, sidebar: Ginther et al 2018 has a little comment that I probably missed originally. “In results not reported, being a full professor increased the probability of NIH funding by 11.9 ppt (p < .001), but the funding gap remained -12.8 ppt (p < .001).” Yeah, age doesn’t help. ] When we are talking 5, 10, 25 applications it is maybe easy to overlook the sweat that goes into making new and credible proposals. Sure, some can be revised proposals and some are retreads of various prior proposals. But they take work and sweat to make them competitive. You are not going to enjoy the NIH-wide average hit rate with consistently below-average proposals!

This brings me back to a related issue that appeared in the Ginther et al 2011 report. “Applications from black and Asian investigators were significantly less likely to receive R01 funding compared with whites for grants submitted once or twice……black and Asian investigators are less likely to be awarded an R01 on the first or second attempt, blacks and Hispanics are less likely to resubmit a revised application, and black investigators that do resubmit have to do so more often to receive an award.“. I will admit I still don’t quite understand what they are presenting here at the end. It reads as though they are gating on Black investigators who do eventually win an award and do so on revision, not the A0 (this sample was back in the A2-permitted days, iirc). This whole passage, however, can be received as “well, just work a little harder to compensate” and to appear as if we’re only talking about an extra revision or two. I probably received this in this way myself on initially seeing the 2011 paper. And I have to say the “1.7 fold advantage” that is discussed in Hoppe for the per-application success rates comes in the same way. It can be received as, well, you just have to write two to get one. Because it focuses us on the “given you did get an award” instead of what it takes to get that award, statistically speaking.

But looking at these cumulative probability graphs really hits differently.

Black PIs don’t have to work just a little harder.

Black PIs have to work a LOT harder.

End of Year Funds in 2020

As you know, the end of the Federal fiscal year can be a fun time for hopeful NIH grant applicants. This is when your favorite ICs are counting up the beans and making sure they use all of their appropriated money.

This means that they often pick up some grants with scores that otherwise looked like they were not going to fund.

This is great if you are one of the lucky ones. I have had 9/30 grant starts in the past. It feels awesome.

This year however….this year.

I am thinking about the Hoppe finding and the original Ginther report. I am thinking as always about the NIH’s complete and utter failure to address this issue. And I am hopeful. Always. That IC Directors will take it upon themselves to follow the advice I had right from the start.

Just FIX this. It won’t take much. Just a few extra grant pickups that happen to have Black PIs. End of year is a great time to slip one or three or five into the portfolio. Nobody can complain about these decisions.

So… pull up RePORTER. Click the start date of 9/15/2020, enter the two letter code for your favorite ICs and start searching.

See any of your Black colleagues getting grants?

NIH grant application topics by IC

As you will recall, the Hoppe et al. 2019 report [blogpost] both replicated Ginther et al 2011 with a subsequent slice of grant applications, demonstrating that after the news of Ginther, with a change in scoring procedures and changes in permissible revisions, applications with Black PIs still suffered a huge funding disparity. Applications with white PIs are 1.7 times more likely to be funded. Hoppe et al also identified a new culprit for the funding disparity to applications with African-American / Black PIs. TOPIC! “Aha”, they crowed, “it isn’t that applications with Black PIs are discriminated against on that basis, no. It’s that the applications with Black PIs just so happen to be disproportionately focused on topics that just so happen to have lower funding / success rates”. Of course it also was admitted very quietly by Hoppe et al that:

WH applicants also experienced lower award rates in these clusters, but the disparate outcomes between AA/B and WH applicants remained, regardless of whether the topic was among the higher- or lower-success clusters (fig. S6).

Hoppe et al., Science Advances, 2019 Oct 9;5(10):eaaw7238. doi: 10.1126/sciadv.aaw7238

If you go to the Supplement Figure S6 you can see that for each of the five quintiles of topic clusters (ranked by award rates) applications with Black PIs fare worse than applications with white PIs. In fact, in the least-awarded quintile, which has the highest proportion of the applications with Black PIs, the white PI apps enjoy a 1.87 fold advantage, higher than the overall mean of the 1.65 fold advantage.

Record scratch: As usual I find something new every time I go back to one of these reports on the NIH funding disparity. The overall award rate disparity was 10.7% for applications with Black PIs versus 17.7% for those with white PIs. The take away from Hoppe et al. 2019 is reflected in the left side of Figure S6 where it shows that the percentage of applications with Black PIs is lowest (<10%) in the topic domains with the highest award rates and highest (~28%) in the domains with the lowest award rates. The percentages are more similar for apps with white PIs, approximately 20% per quintile. But the right side lists the award rates by quintile. And here we see that in the second highest award-rate topic quintile, the disparity is similar to the mean (12.6% vs 18.9%) but in the top quintile it is greater (13.4% vs 24.2% or a 10.8%age point gap vs the 7%age point gap overall). So if Black PIs followed Director Collins’ suggestion that they work on the right topics with the right methodologies, they would fare even worse due to the 1.81 fold advantage for applications with white PIs in the top most-awarded topic quintile!

Okay but what I really started out to discuss today was a new tiny tidbit provided by a blog post on the Open Mike blog. It reports the topic clusters by IC. This is cool to see since the word clusters presented in Hoppe (Figure 4) don’t map cleanly onto any sort of IC assumptions.

https://nexus.od.nih.gov/all/2020/08/12/institute-and-center-award-rates-and-funding-disparities/

All we are really concerned with here is the ranking along the X axis. From the blog post:

…17 topics (out of 148), representing 40,307 R01 applications, accounted for 50% of the submissions from African American and Black (AAB) PIs. We refer to these topics as “AAB disproportionate” as these are topics to which AAB PIs disproportionately apply.

Note the extreme outliers. One (MD) is the National Institute on Minority Health and Health Disparities. I mean… seriously. The other (NR) is the National Institute on Nursing Research which is also really interesting. Did I mention that these two Is get 0.8% and 0.4% of the NIH budget, respectively? The NIH mission statement reads: “NIH’s mission is to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability.” Emphasis added. The next one (TW) is the Fogerty International Center which focuses on global health issues (hello global pandemics!) and gets 0.2% of the NIH budget.

Then we get into the real meat. At numbers 4-6 on the AAB Disproportionate list of ICs we reach the National Institute on Child Health and Development (HD, 3.7% of the budget), NIDA (DA, 3.5%) and NIAAA (AA, 1.3%). And clocking in at 7 and 9 we have National Institute on Aging (AG, 8.5%) and the NIMH (MH, 4.9%).

These are a lot of NIH dollars being expended in ICs of central interest to me and a lot of my audience. We could have made some guesses based on the word clusters in Hoppe et al 2019 but this gets us closer.

Yes, we now need to get deeper and more specific. What is the award disparity for applications with Black vs white PIs within each of these ICs? How much of that disparity, if it exists, accounted for by the topic choices within IC?

And lets consider the upside. If, by some miracle, a given IC is doing particularly well with respect to funding applications with Black PIs fairly….how are they accomplishing this variance from the NIH average? What can the NIH adopt from such an IC to improve things?

Oh, and NINR and NIMHHD really need a boost to their budgets. Maybe NIH Director Collins could put a 10% cut prior to award to the other ICs to improve investment in the applying-knowledge-to-enhance-health goals of the mission statement?