I have long standing doubts about certain aspects of funding mechanisms that are targeted to underrepresented individuals. This almost always has come up in the past in the context of graduate or postdoctoral fellowships and when there is a FOA open to all, and a related or parallel FOA that is directed explicitly at underrepresented individuals. For example see NINDS F31, K99/R00 , NIGMS K99/R00 initiatives, and there is actually a NIH parent F32 – diversity as well).

At first blush, this looks awesome! Targeted opportunity, presumably grant panel review that gives some minimal attention to the merits of the FOA and, again presumably, some Program traction to fund at least a few.

My Grinchy old heart is, however, suspicious about the real opportunity here. Perhaps more importantly, I am concerned about the real opportunity versus the opportunity that might be provided by eliminating any disparity of review that exists for the review of applications that come in via the un-targeted FOA. No matter the FOA, the review of NIH grants is competitive and zero sum. Sure, pools of money can be shifted from one program to another (say from the regular F31 to the F31-diversity) but it is rarely the case there is any new money coming in. Arguing about the degree to which funding is targeted by decision of Congress, of the NIH Director, of IC Directors or any associated Advisory Councils is a distraction. Sure NIGMS gets a PR hit from announcing and funding some MOSAIC K99/R00 awards…but they could just use those moneys to fund the apps coming in through their existing call that happen to have PIs who are underrepresented in science.

The extreme example here is the highly competitive K99 application from a URM postdoc. If it goes in to the regular competition, it is so good that it wins an award and displaces, statistically, a less-meritorious one that happens to have a white PI. If it goes in to the MOSAIC competition, it also gets selected, but in this case by displacing a less-meritorious one that happens to have a URM PI. Guaranteed.

These special FOA have the tendency to put all the URM in competition with each other. This is true whether they would be competitive against the biased review of the regular FOA or, more subtly, whether they would be competitive for funding in a regular FOA review that had been made bias-free(r).

I was listening to a presentation from Professor Nick Gilpin today on his thoughts on the whole Ginther/Hoppe situation (see his Feature at eLife with Mike Taffe) and was struck by comments on the Lauer pre-print. Mike Lauer, head of NIH’s office of extramural awards, blogged and pre-printed an analysis of how the success rates at various NIH ICs may influence the funding rate for AA/B PIs. It will not surprise you that this was yet another attempt to suggest it was AA/B PIs’ fault that they suffer a funding disparity. For the sample of grants reviewed by Lauer (from the Hoppe sample), 2% were submitted with AA/B PIs, NIH-wide. The percentage submitted to the 19 individual funding ICs he covered ranged from 0.73% to 14.7%. This latter institute was the National Institute on Minority Health and Health Disparities (NIMHD). Other notable ICs of disproportionate relevance to the grants submitted with AA/B PIs include NINR (4.6% AA/B applications) and NICHD (3%).

So what struck me, as I listened to Nick’s take on these data, is that this is the IC assignment version of the targeted FOA. It puts applications with AA/B investigators in higher competition with each other. “Yeahbutt”, you say. It is not comparable. Because there is no open competition version of the IC assignment.

Oh no? Of course there is, particularly when it comes to NIMHD. Because these grants will very often look like a grant right down the center of those of interest to the larger, topic-focused ICs….save that it is relevant to a population considered to be minority or suffering a health disparity. Seriously, go to RePORTER and look at new NIMHD R01s. Or heck, NIMHD is small enough you can look at the out year NIMHD R01s without breaking your brain since NIHMH only gets about 0.8% of the NIH budget allocation. With a judicious eye to topics, some related searches across ICs, and some clicking on the PI names to see what else they may have as funded grants, you can quickly convince yourself that plenty of NIMHD awards could easily be funded by a related I or C with their much larger budgets*. Perhaps the contrary is also true, grants funded by the parent / topic IC which you might also argue would fit at NIMHD, but I bet the relative percentage goes the first way.

If I am right in my suspicions, the existence of NIMHD does not necessarily put more aggregate money into health disparities research. That is, more than that which could just as easily come out of the “regular” budget. The existence of NIMHD means that the parent IC can shrug off their responsibility for minority health issues or disparity issues within their primary domains of drug abuse, cancer, mental health, alcoholism or what have you. Which means they are likewise shrugging off the AA/B investigators who are disproportionately submitting applications with those NIMHD-relevant topics and being put in sharp competition with each other. Competition not just within a health domain, but across all health domains covered by the NIH.

It just seems to me that putting the applications with Black PIs preferentially in competition with themselves, as opposed to making it a fair competition for the entire pool of money allocated to the purpose, is sub optimal.

__

*Check out the descriptions for MD010362 and CA224537 for some idea of what I mean. The entire NIMHD budget is 5% as large as the NCI budget. Why, you might ask, is NCI not picking up this one as well?

This is not news to this audience but it bears addressing in as many ways as possible, in the context of the Hoppe et al 2019 and Ginther et al 2011 findings. Behind most of the resistance to doing anything about the funding disparity for investigators and, as we’re now finding out, topics is still some lingering idea that the NIH grant selection process is mostly about merit.

Objective merit. Sure we sort of nod that we understand that there is some wiggle room but overall it is difficult to find anyone that appears to understand something in a deep way.

“Merit” of NIH grants is untethered to anything objective. It relies on the opinion of the peer reviewers. The ~3 reviewers who are assigned to do deep review and the members of the panel (which can be 20-30ish folks) who vote scores after the discussion.

This particular dumb twitter poll shows that 77% of experienced reviewers either occasionally or regularly have the experience of thinking a grant that should not receive funding is very likely to do so.

and this other dumb little twitter poll shows that 88% of experienced NIH grant reviewers either occasionally or frequently experience a panel voting a non-fundable score for a grant they think deserves funding.

It will not escape you that individual reviewers tend to think a lot more grants should be funded than can be funded. And this shows up in the polls to some extent.

This is not high falutin’ science and it is possible we have some joker contamination here from people who are not in fact NIH review experienced.

But with that caveat, it tends to support the idea that the mere chance of which individuals are assigned to review a grant can have a major effect on merit. After all, the post-discussion scores of these individuals tends to significantly constrain the voting. But the voting is important too, since panel members can go outside the range or decide en masse to side with one or the other ends of the post-discussion range.

Swap out the assigned reviewers for a different set of three individuals and the outcomes are likely to be very different. Swap out one panel for another and the tendencies could be totally different. Is your panel heavy in those interested in sex differences and/or folks heavily on board with SABV? Or is it dominated by SABV resisters?

Is the panel super interested in the health effects of cannabis and couldn’t give a fig about methamphetamine? What do YOU think is going to come out of that panel with fundable scores?

Does the panel think any non-mammalian species is horrible for modeling human health and should really never be funded? Does the panel geek away at tractable systems and adore anything fly or worm driven and complain about the lack of manipulability available in a rat?

Of course you know this. These kinds of whines and complaints are endemic to fireside chats whenever two or more NIH grant-seeking investigators are present!

But somehow when it is a disparity of race or of topics of interest to minority communities in the US, such as from Hoppe et al 2019, then nobody is concerned. Even when there are actual data on the table showing a funding disparity. And everyone asks their “yeahbutwhatabout” questions, springing right back into the mindset that at the very core the review and selection of grants is about merit. The fact their worm grant didn’t get selected is clear evidence of a terrible bias in the NIH approach. The fact African-American PIs face a payline far lower than they do…..snore.

Because in that case it is about objective merit.

And not about the coincidence of whomever the SRO has decided should review that grant.

On his first day in office President Biden signed an Executive Order described thusly by the NYT. I am having difficulty finding a link to the exact text right now. [Edited to Add: The Executive Order On Advancing Racial Equity and Support for Underserved Communities Through the Federal Government.]

“The president designated Susan E. Rice, who is the head of his Domestic Policy Council, as the leader of a “robust, interagency” effort requiring all federal agencies to make “rooting out systemic racism” central to their work. His order directs the agencies to review and report on equity in their ranks within 200 days, including a plan on how to remove barriers to opportunities in policies and programs. The order also moves to ensure that Americans of all backgrounds have equal access to federal government resources, benefits and services. It starts a data working group as well as the study of new methods to measure and assess federal equity and diversity efforts.”

Well, the NIH doesn’t have to take 200 days to “review and report”. They’ve already done so in

Ginther, D.K., Schaffer, W.T., Schnell, J., Masimore, B., Liu, F., Haak, L.L., Kington, R., 2011. Race, ethnicity, and NIH research awards. Science 333(6045), 1015-1019.

and

Hoppe, T.A., Litovitz, A., Willis, K.A., Meseroll, R.A., Perkins, M.J., Hutchins, B.I., Davis, A.F., Lauer, M.S., Valantine, H.A., Anderson, J.M., Santangelo, G.M., 2019. Topic choice contributes to the lower rate of NIH awards to African-American/black scientists. Sci Adv 5(10), eaaw7238.

So we can skip the 200 days worth of can kicking, Dr. Collins, and move straight to the fixing part. The “ensure” part. The “equal access” part.

This means funding research on topics that are important to Americans of all backgrounds, including African-American ones. Equally. This means pumping up the budget of the National Institute on Minority Health and Health Disparities (NIMHD). It also means holding the other ICs responsible for taking on their share of these projects and not just shrugging them off into NIMHD.

It means funding not just white American science teams that work on these topics but funding teams of African-American investigators. Equally.

It also means not just funding African-American professors to work on topics of relevance to the health interests of African-Americans but rather equalizing the funding chances of African-American PIs who choose to work on any topic at all.

It’s time to go big. Forces within the NIH who have been trying to do good on this should feel empowered to shout down the nay sayers and to hold the foot draggers to account.

Forces outside the NIH who have been trying to do good on this should likewise feel empowered to hold Susan Rice, their Senators and Congress Reps to account.

The NIH has launched a new FOA called the Stephen I. Katz Early Stage Investigator Research Project Grant (Open Mike blog post). PAR-21-038 is the one for pre-clinical, PAR-21-039 is the one for clinical work. These are for Early Stage Investigators only and have special receipt dates (e.g. January 26, 2021; May 26, 2021; September 28, 2021). Details appear to be a normal R01- up to 5 years and any budget you want to try (of course over $500k per year requires permission).

The novelty here appears to be entirely this:

For this FOA, applications including preliminary data will be considered noncompliant with the FOA instructions and will be withdrawn. Preliminary data are defined as data not yet published. Existence of preliminary data is an indication that the proposed project has advanced beyond the scope defined by this program and makes the application unsuitable for this funding opportunity. Publication in the proposed new research direction is an indication that the proposed work may not be in a new research direction for the ESI.

This will be fascinating. A little bit more specification that the scientific justification has to rest on published (or pre-printed) work only:

The logical basis and premise for the proposed work should be supported by published data or data from preprints that have a Digital Object Identifier (DOI). These data must be labeled and cited adjacent to each occurrence within the application and must be presented unmodified from the original published format. Figures and tables containing data must include citation(s) within the legend. The data should be unambiguously identified as published through citation that includes the DOI (see Section IV.2). References and data that do not have an associated DOI are not allowed in any section of the application. Prospective applicants are reminded that NIH instructions do not allow URLs or hyperlinks to websites or documents that contain data in any part of the application

So how is this going to work in practice for the intrepid ESI looking to apply for this?

First, there is no reason you have to put the preliminary data you have available in the application. One very hot comment over at the Open Mike blog post about the proposals being unsupported and therefore the projects will be doomed to failure is totally missing this point. PIs are not stupid. They aren’t going to throw up stupid ideas, they are going to propose their good ideas that can be portrayed as being unsupported by preliminary data.

Twill be interesting to see how this is interpreted vis a vis meeting presentations, seminars and (hello!) job talks. What is a reviewer expected to do if they see an application without any preliminary data per the FOA, but have just seen a relevant presentation from the applicant which shows that Aim 1 is already completed? Will they wave a flag? See above, the FOA says the “existence” of preliminary data, not the “inclusion” of preliminary data will make the app non-compliant.

But there is an aspect of normal NIH grant review that is not supposed to depend on “secret” knowledge, i.e., that available only to the reviewer, not published. So it is frowned upon for a reviewer to say “well the applicant gave a seminar last month at our department and showed that this thing will work”. It’s special knowledge only available to that particular reviewer on the panel. Unverifiable.

This would be similar, no?

Or is this more like individual knowledge that the PI had faked data? In such cases the reviewers are obligated to report that to the SRO in private but not to bring it up during the review.

If they ARE going to enforce the “existence” of relevant preliminary data, how will it be possible to make this fair? It will be entirely unfair. Some applicants will be unlucky enough to have knowledgeable whistle blowers on the panel and some will evade that fate by chance. Reviewers being what they are, will only variably respond to this charge to enforce the preliminary data thing, even if semi-obligated. After all, what is the threshold for the data being specifically supportive of the proposal at hand?

Strategy-wise, of course I endorse ESI taking advantage of this. The FOAs list almost all of the ICs with relevant funding authority if I counted correctly (fewer for the human-subjects one, of course). There is an offset receipt date, so it keeps the regular submission dates clear. You can put one in, keep working on it and if the prelim data look good, put a revised version in for a regular FOA next time. Or, if you can’t work on it or the data aren’t going well, you can resubmit “For Resubmissions, the committee will evaluate the application as now presented, taking into consideration the responses to comments from the previous scientific review group and changes made to the project.” Win-win.

Second strategy thing. This is a PAR and the intent is to convene panels for this mechanism. This means that your relative ESI advantage at the point of review disappears. You are competing only against other ESI. Now, how each IC chooses to prioritize these is unknown. But once you get a score, you are presumably just within whatever ESI policy a given IC has set for itself.

I’m confused by the comments over at Open Mike. They seem sort of negative about this whole thing. It’s just another FOA, folks. It doesn’t remove opportunities like the R15. No it doesn’t magically fix every woe related to review. It is an interesting attempt to fix what I see as a major flaw in the evolved culture of NIH grant review and award. Personally I’d like to see this expanded to all applicants but this is a good place to start.

Happy 2021!!!!!

January 1, 2021

As we turn our backs on 2020, a real jackass of a year, I wish you all good fortune. May your grant applications be funded, your papers be published and your exciting new science keep you jumping to see new data every week.

I’m not a big fan of huge sweeping goals and resolutions, personally. And right now, I am going to be satisfied with putting my research program back together.

As you will recall, I changed jobs in early 2019. We were juuuuusst getting the laboratory into something resembling operational shape in January 2020. And then Covid hit.

This has been very much not-fun for me. As you can imagine. The hardest thing of all is the loss of the data stream. As I may have mentioned before this is the bulk of the reason why I do this job. To see the data.

But I also picked up some new responsibilities in 2020, in part due to the job change and in part due to the death of George Floyd, murdered by Minneapolis police officers in May. Our various academic institutions had a bit of a moment. This caused an opening. A “strike while the iron is hot” moment. From my perspective anyway. So, despite a certain weariness with institutional efforts on diversity, I’m back in the fight. I say yes to way more things than I would have prior to May of 2020. I am accepting more obvious tokenism offers. I am bearing down.

I plan to continue that for a little bit more. Until the steam seems to have escaped and the ingot as cold as the earth.

I have been delighted to see many of you doing the same. Recognizing this moment and getting down to work with hammer and tongs. I know you can’t all sustain these efforts forever. Do it as long as you can. And then rest knowing you did your part.

Happy New Year.