NIH data on Discussion Rates and Grant Submitting Vigor

The latest blog post over at Open Mike, from the NIH honcho of extramural grant award Mike Lauer, addresses “Discussion Rate”. This is, in his formulation, the percent of applicants (in a given Fiscal Year, FY21 in this case) who are PI on at least one application that reaches discussion. I.e., not triaged. The post presents three Tables, with this Discussion rate (and Funding rate) presented by the Sex of the PI, by race (Asian, Black, White only) or ethnicity (Hispanic or Latino vs non-Hispanic only). The tables further presented these breakdowns by Early Stage Investigator, New Investigator, At Risk and Established. At risk is a category of “researchers that received a prior substantial NIH award but, as best we can tell, will have no funding the following fiscal year if they are not successful in securing a competing award this year.” At this point you may wish to revisit an old blog post by DataHound called “Mind the Gap” which addresses the chances of regaining funding once a PI has lost all NIH grants.

I took the liberty of graphing the By-Race/Ethnicity Discussion rates, because I am a visual thinker.

There seem to be two main things that pop out. First, in the ESI category, the Discussion rate for Black PI apps is a lot lower. Which is interesting. The 60% rate for ESI might be a little odd until you remember that the burden of triage may not fall on ESI applications. At least 50% have to be discussed in each study section, small numbers in study section probably mean that on average it is more than half, and this is NIH wide data for FY 21 (5,410 ESI PIs total). Second, the NI category (New, Not Early on the chart) seems to suffer relative to the other categories.

Then I thought a bit about this per-PI Discussion rate being north of 50% for most categories. And that seemed odd to me. Then I looked at another critical column on the tables in the blog post.

The Median number of applications per applicant was…. 1. That means the mode is 1.

Wow. Just….wow.

I can maybe understand this for ESI applicants, since for many of them this will be their first grant ever submitted.

but for “At Risk”? An investigator who has experience as a PI with NIH funding, is about to have no NIH funding if a grant does not hit, and they are submitting ONE grant application per fiscal year?

I am intensely curious how this stat breaks down by deciles. How many at risk PIs are submitting only one grant proposal? Is it only about half? Two-thirds? More?

As you know, my perspective on the NIH grant getting system is that if you have only put in one grant you are not really trying. The associated implication is that any solutions to the various problems that the NIH grant award system might have that are based on someone not getting their grant after only one try are not likely to be that useful.

I just cannot make this make sense to me. Particularly if the NIH

It is slightly concerning that the NIH is now reporting on this category of investigator. Don’t get me wrong. I believe this NIH system should support a greater expectation of approximately continual funding for investigators who are funded PIs. But it absolutely cannot be 100%. What should it be? I don’t know. It’s debatable. Perhaps more importantly who should be saved? Because after all, what is the purpose of NIH reporting on this category if they do not plan to DO SOMETHING about it? By, presumably, using some sort of exception pay or policy to prevent these at risk PIs from going unfunded.

There was just such a plan bruited about for PIs funded with the ESI designation that were unable to renew or get another grant. They called them Early Established Investigators and described their plans to prioritize these apps in NOT-OD-17-101. This was shelved (NOT-OD-18-214) because “NIH’s strategy for achieving these goals has evolved based on on-going work by an Advisory Committee to the Director (ACD) Next Generation Researchers Initiative Working Group and other stakeholder feedback” and yet asserted “NIH..will use an interim strategy to consider “at risk investigators”..in its funding strategies“. In other words, people screamed bloody murder about how it was not fair to only consider “at risk” those who happened demographically to benefit from the ESI policy.

It is unclear how these “consider” decisions have been made in the subsequent interval. In a way, Program has always “considered” at risk investigators, so it is particularly unclear how this language changes anything. In the early days I had been told directly by POs that my pleas for an exception pay were not as important because “we have to take care of our long funded investigators who will otherwise be out of funding”. This sort of thing came up in study section more than once in my hearing, voiced variously as “this is the last chance for this PIs one grant” or even “the PI will be out of funding if…”. As you can imagine, at the time I was new and full of beans and found that objectionable. Now….well, I’d be happy to have those sentiments applied to me.

There is a new version of this “at risk” consideration that is tied to the new PAR-22-181 on promoting diversity. In case you are wondering why this differs from the famously rescinded NINDS NOSI, well, NIH has managed to find themselves a lawyered excuse.

Section 404M of the Public Health Service Act (added by Section 2021 in Title II, Subtitle C, of the 21st Century Cures Act, P.L. 114-255, enacted December 13, 2016), entitled, “Investing in the Next Generation of Researchers,” established the Next Generation Researchers Initiative within the Office of the NIH Director.  This initiative is intended to promote and provide opportunities for new researchers and earlier research independence, and to maintain the careers of at-risk investigators.  In particular, subsection (b) requires the Director to “Develop, modify, or prioritize policies, as needed, within the National Institutes of Health to promote opportunities for new researchers and earlier research independence, such as policies to increase opportunities for new researchers to receive funding, enhance training and mentorship programs for researchers, and enhance workforce diversity;

“enacted December 13, 2016“. So yeah, the NOSI was issued after this and they could very well have used this for cover. The NIH chose not to. Now, the NIH chooses to use this aspect of the appropriations language. And keep in mind that when Congress includes something like this NGRI in the appropriations language, NIH has requested it or accepted it or contributed to exactly how it is construed and written. So this is yet more evidence that their prior stance that the “law” or “Congress” was preventing them from acting to close the Ginther Gap was utter horseshit.

Let’s get back to “at risk” as a more explicitly expressed concern of the NIH. What will these policies mean? Well, we do know that none of this comes with any concrete detail like set aside funds (the PAR is not a PAS) or ESI-style relaxation of paylines. We do know that they do this all the damn time, under the radar. So what gives? Who is being empowered by making this “consideration” of at-risk PI applications more explicit? Who will receive exception pay grants purely because they are at risk? How many? Will it be in accordance with distance from payline? How will these “to enhance diversity” considerations be applied? How will these be balanced against regular old “our long term funded majoritarian investigator is at risk omg” sentiments in the Branches and Divisions?

This is one of the reasons I like the aforementioned Datahound analysis, because at least it gave a baseline of actual data for discussion purposes. A framework a given I or C could follow in starting to make intelligent decisions.

What is the best policy for where, who, what to pick up?

Research colonialism, analyzing an RFA

I recently fielded a question from a more junior scientist about what, I think, has been termed research colonialism with specificity to the NIH funding disparity known as the Ginther Gap. One of the outcomes of the Hoppe et al 2019 paper, and the following Lauer et al 2021, was a call for a hard look at research on the health issues of communities of color. How successful are grant proposals on those topics, which ICs are funding them, what are the success rates and what are the budget levels appropriated to, e.g. the NIMHD. I am very much at sea trying to answer the question I was asked, which boiled down to “Why is it always majoritarian PIs being funded to do research with communities of color?”. I really don’t know how to answer that or how to begin to address it with NIH funding data that has been generated so far. However, something came across my transom recently that is a place to start.

The NIH issued RFA-MD-21-004 Understanding and Addressing the Impact of Structural Racism and Discrimination on Minority Health and Health Disparities last year and the resulting projects should be on the RePORTER books by now. I was cued into this by a tweet from the Constellation Project which is something doing co-author networks. That may be useful for a related issue, that of collaboration and co-work. For now, I’m curious about what types of PIs have been able to secure funding from this mechanism. According to my RePORTER search for the RFA, there are currently 17 grants funded.

Of the funded grants, there are 4 from NIMHD, 4 from NIDA, 2 from NIA, 1 each from NIMH, NIHNDS, NINR, NICHD, NIGMS, NIDCD, and NCCIH. In the RFA, NIMHD promised 6-7 awards, NIDA 2, NIA 6, NIGMS 4-6 so obviously NIDA overshot their mark, but the rest are slacking. One each was promised for NIMH, NINDS, NICHD, NIDCD and NCCIH, so all of these are on track. Perhaps we will see a few more grants get funded by the time the FY elapses on Sept 30.

So who is getting funded under this RFA? Doing a quick google on the PIs, and admittedly making some huge assumptions based on the available pictures, I come up with

PI/Multi-PI Contact: White woman (2 NIA; 1 NCCIH; 3 NIDA; 1 NIDCD; 1 NIGMS; 1 NINDS); Black woman (1 NIDA; 1 NICHD; 1 NIMHD); Asian woman (1 NIMHD; 1 NIMHD; 1 NINR); White man (1 NIMHD; 1 NIMH)

Multi-PI, non-contact: Asian woman (1 NIA, 1 NIDA, 1 NIMHD); Black woman (2 NIDA, 1 NIMHD); White woman (1 NIDCD; 1 NIGMS; 1 NINR) Black man (1 NIGMS; 1 NIMH); White man (2 NIMH)

I would say the place I am most likely to be off in terms of someone who appears to me to be white but identifies as a person of color would be white women. Maybe 2-3 I am unsure of. I didn’t bother to keep track of how many of the non-contact PIs are on the proposals with white Contact PIs versus the other way around but….I can’t recall seeing even one where a non-contact white PI was on a proposal with a contact PI who is Black or Asian. (There was one award with three white men and one Black man as PIs and, well, does anyone get away with a four PI list that includes no woman anymore?) Anyway… make of that what you will.

I suspect that this RFA outcome is probably slightly better than the usual? And that if you looked at NIH’s studies that deal with communities or color and/or their health concerns more generally it would be even more skewed towards white PIs?

Ginther et al 2011 reported 69.9% of apps in their sample had white PIs, 16.2% had Asian PIs and 1.4% had Black PIs. Hoppe et al 2019 reported (Table S1) 1.5% of applications had Black PIs and 65.7% had white PIs in their original sample. So the 11 out of 17 grants having white PIs/Contact MultiPIs matches expected distribution, as does 3 Asian PIs. Black PIs are over represented since 1-2% of 17 is..zero grants funded. So this was not an opportunity that NIH took to redress the Ginther Gap.

But should it be? What should be the identity of PIs funded to work on issues related to “racism and discrimination” as it applies to “minority health and health disparities”? The “best” as determined by a study section of peer scientists, regardless of applicant characteristics? Regardless of the by now very well established bias against applications with Black PIs?

Someone on twitter asked about the panel that reviewed these grants. You can see from the funded grants on RePORTER that the study section reviewing these proposals was ZMD1 KNL (J1). Do a little web searching and you find that the roster for the 11/15/2021-11/17/2021 meeting is available. A three day meeting. That must have been painful. There are four chairs and a huge roster listed. I’m not going to search out all of them to figure out how many were white on the review panel. I will note that three of the four chairs were white and one was Asian (three of four were MDs, one was a PHD). This is a good place for a reminder that Hoppe et al reported 2.4% of reviewers were Black and 77.8% white in the study sections reviewing proposals for funding in FY2011-2015. I would be surprised if this study section was anything other than majority white.

Research Opportunities for New and “At-Risk” Investigators to Promote Workforce Diversity

NIDA, NIMH, and NINDS have issued a Program Announcement (PAR-22-181) to provide Research Opportunities for New and “At-Risk” Investigators with the intent to Promote Workforce Diversity.

This is issued as a PAR, which is presumably to allow Special Emphasis Panels to be convened. It is not a PAS, however, the announcement includes set-aside funding language familiar to PAS and RFA Funding Opportunity Announcements (FOA).

Funds Available and Anticipated Number of Awards The following NIH components intend to commit the following amounts for the duration of this PAR: NINDS intends to commit up to $10 million per fiscal year, approximately 25 awards, dependent on award amounts; NIDA intends to commit up to $5 million per fiscal year, 12-15 awards, dependent on award amounts; NIMH intends to commit up to $5 million per fiscal year, 12-15 awards, dependent on award amounts; Future year amounts will depend on annual appropriations.

This is a PA typical 3 year FOA which expires June 7, 2025. Reciept dates are one month ahead of standard, i.e., Sept (new R01) / Oct (Resub, Rev, Renew); Jan/Feb; May/Jun for the respective Cycles.

Eligibility is in the standard categories of concern including A) Underrepresented Racial/Ethnic groups, B) Disability, C) economic disadvantage and D) women. Topics of proposal have to be within the usual scope of the participating ICs. Eligibility of PIs is for the familiar New Investigators (“has not competed successfully for substantial, NIH (sic) independent funding from NIH“) and a relatively new “at risk” category.

At risk is defined as “has had prior support as a Principal Investigator on a substantial independent research award and, unless successful in securing a substantial research grant award in the current fiscal year, will have no substantial research grant funding in the following fiscal year.“

So. We have an offset deadline (at least for new proposals), set aside funds, SEPs for review and inclusion of NI (instead of merely ESI) and the potential for the more experienced investigator who is out of funding to get help as well. Pretty good! Thumbs up. Can’t wait to see other ICs jump on board this one.

To answer your first question, no, I have no idea how this differs from the NINDS/NIDA/NIAAA NOSI debacle. As a reminder:

Notice NOT-NS-21-049 Notice of Special Interest (NOSI): NIH Research Project Grant (R01) Applications from Individuals from Diverse Backgrounds, Including Under-Represented Minorities was released on May 3, 2021.

The “debacle” part is that right after NIDA and NIAAA joined NINDS in this NOSI, the Office of the Director put it about that no more ICs could join in and forced a rescinding of the NOSI on October 25, 2021 while claiming that their standard issue statement on diversity accomplished the same goals.

I see nothing in this new PAR that addresses either of the two real reasons that may have prompted the Office of the Director to rescind the original NOSI. The first and most likely reason is NIH’s fear of right wing, anti-affirmative action, pro-white supremacy forces in Congress attacking them. The second reason would be people in high places* in the NIH that are themselves right wing, anti-affirmative action and pro-white supremacy. If anything, the NOSI was much less triggering since it came with no specific plans of action or guarantees of funding. The PAR, with the notification of intended awards, is much more specific and would seemingly be even more offensive to right wingers.

I do have two concerns with this approach, as much as I like the idea.

First, URM-only opportunities have a tendency to put minority applicants in competition with each other. Conceptually, suppose there is an excellent URM qualified proposal that gets really high priority scores from study section and presume it would have also done so in an open, representation-blind study section. This one now displaces another URM proposal in the special call and *fails to displace* a lesser proposal from (statistically probable) a majoritarian PI. That’s less good than fixing the bias in the first place so that all open competitions are actually open and fair. I mentioned this before:

These special FOA have the tendency to put all the URM in competition with each other. This is true whether they would be competitive against the biased review of the regular FOA or, more subtly, whether they would be competitive for funding in a regular FOA review that had been made bias-free(r). […] The extreme example here is the highly competitive K99 application from a URM postdoc. If it goes in to the regular competition, it is so good that it wins an award and displaces, statistically, a less-meritorious one that happens to have a white PI. If it goes in to the MOSAIC competition, it also gets selected, but in this case by displacing a less-meritorious one that happens to have a URM PI. Guaranteed.

The second concern is one I’ve also described before.

In a news piece by Jocelyn Kaiser, the prior NIH Director Elias Zerhouni was quoted saying that study sections responded to his 2006/2007 ESI push by “punishing the young investigators with bad scores”. As I have tried to explain numerous times, phrasing this as a matter of malign intent on the part of study section members is a mistake. While it may be true that many reviewers opposed the idea that ESI applicants should get special breaks, adjusting scores to keep the ESI application at the same chances as before Zerhouni’s policies took effect is just a special case of a more general phenomenon.

So, while this PAR is a great tactical act, we must be very vigilant for the strategic, long term concerns. It seems to me very unlikely that there will be enthusiasm for enshrining this approach for decades (forever?) like the ESI breaks on merit scores/percentiles/paylines. And this approach means it will not be applied by default to all qualifying applications, as is the case for ESI.

Then we get to the Oppression Olympics, an unfortunate pitting of the crabs in the barrel against each other. The A-D categories of under-representation and diversity span quite a range of PIs. People in each category, or those who are concerned about specific categories, are going to have different views on who should be prioritized. As you are well aware, Dear Reader, my primary concern is with the Ginther gap. As you are aware, the “antis” and some pro-diversity types are very concerned to establish that a specific person who identifies as African-American has been discriminated against and is vewwwwy angweee to see any help being extended to anyone of apparent socio-economic privileges who just so happens to be Black. Such as the Obama daughters. None of us are clean on this. Take Category C. I have relatively recently realized that I qualify under Category C since I tick three of the elements, only two are required. I do not think that there is any possible way that my qualification on these three items affects my grant success in the least. To do so would require a lot of supposing and handwaving. I don’t personally think that anyone like me who qualifies technically under Category C really should be prioritized against, say, the demonstrated issue with the Ginther gap. These are but examples of the sort of “who is most disadvantaged and therefore most deserving” disagreement that I think may be a problem for this approach.

Why? Because reviewers will know that this is the FOA they are reviewing under. Opinions on the relative representation of categories A-D, Oppression Olympics and the pernicious stanning of “two-fers” will be front and present. Probably explicit in some reviews. And I think this is a problem in the broader goals of improving equity of opportunity and in playing for robust retention of individuals in the NIH funded research game.

__

*This is going to have really ugly implications for the prior head of the NIH, Francis Collins, if the PAR is not rescinded from the top and the only obvious difference here is his departure from NIH.

Grant awards and the new, new NIH Biosketch

Way back in 2015 the NIH made some major changes to the Biosketch. As detailed in this post, one of the major changes was replacing the long list of publications with a “contribution to science” section which was supposed to detail up to five areas of focus with up to four papers, or other research products, cited for each contribution. Some of the preamble from NIH on this suggests it was supposed to be an anti-Glamour measure. Sure. There was also an inclusion of a “personal statement” which was supposed to be used to further brag on your expertise as well as to explain anything…funny… about your record.

In dealing with the “contributions to science” change, I more or less refused to do what was requested. As someone who had been a PI for some time, had mostly senior author pubs and relatively few collaborative papers, I could do this. I just made a few statements about an area I have worked in and listed four papers for each. I didn’t describe my specific role as instructed. I didn’t really describe the influence or application to health or technology. So far this has gone fine, as I can’t remember any comments on Investigator on grants I’ve submitted with this new (old) Biosketch that appear confused about what I have done.

The NIH made some other changes to the Biosketch in 2021, the most notable of which was the removal of the list of Research Support that was previously in Section D. I pointed out in a prior post that I suspect this was supposed to be an attempt to break a specific culture of peer review. One that had hardened reviewers and applicants against the longstanding intent of the NIH. It is very clear in the prior instructions that Section D was not supposed to list all active and completed funding over the past three years. The NIH instructed us to only include that which we wanted to call attention to and added the note that it was for reviewers to assess qualifications of the research team for the new project being proposed. They further underlined this by instructing applicants not to confuse this with the Other Support page which was explicitly for reporting all funding. This failed entirely.

As we have said many times, many ways on this blog…. woe betide any poor newbie applicant who takes the instructions about other grant support at face value and omits any funding that can be easily found on funder websites or the investigator’s lab or University splash page. Reviewers will get in a high dudgeon if they think the PI is trying to conceal anything about their research support. This is, I will assert, because they either overtly or covertly are interested in two things. Neither of which the NIH wants them to be interested in.

One, does the PI have “too much money” in their estimation. The NIH is absolutely opposed to reviewers letting their evaluation of proposal merit be contaminated with such concerns but….people are people and jealously reigns supreme. As does self-righteous feelings about how NIH funds should be distributed. So…review, in practice, is biased in a way that the NIH does not like.

The second concern is related, but is about productivity and is therefore slightly more palatable to some. If the recitation of funding is selective, the PI might be motivated to only present projects that have been the most productive or led to the most Glammy papers. They might be also motivated to omit listing any project which have, by some views, under-produced. This is a tricky one. The instructions say reviewers will look at what the PI chooses to list on the Biosketch as evidence of their overall qualifications. But. How can a reviewer assess qualifications only from the projects that went amazingly well without also assessing how many tanked, relatively speaking? Or so would think a reviewer. The NIH is a little more wobbly on this one. “Productivity” is a sort-of tolerated thing and some analysis of papers-per-grant-dollar (e.g. from NIGMS) show their interest, at least from a Program policy perspective. But I think overall that Program does not want this sort of reviewer bean counting to contaminate merit review too much- the Biosketch instructions insist that the direct costs should not be included for any grants that are mentioned. Program wants to make the calls about “too much money”.

Ok so why am I blogging this again today? Well, we’re into the second year of the new, new attempt of NIH to get the list of grants on the Biosketch more selective. And I’m thinking about how this has been evolving in grants that I’ve been asked to review. Wait..”more selective”? Oh yes, the list of grants can now be added to Section A, the Personal Statement. With all of the same language about how this is only for ongoing or completed projects “that you want to draw attention to“. NOT-OD-21-073 even ties this new format description to the re-organization of the Other Support page, again making it clear that these are not the same thing.

So the question of the day is, how are applicants responding? How are reviewers reacting to various options taken by applicants?

I put in my first few applications with the grant list simply removed. I added a statement to Section A summarizing my total number of intervals of competitive support as PI and left it at that. But I’ve seen many applicants who put all their grants in Section A, just as they would have put them in Section D before.

I guess I had better do the same?

Making sense of the T32 effect on the NIH funding disparity

A new blog post from Mike Lauer, Deputy Director for Extramural Research at the NIH, presents new data on grant award (K and R01equivalent) to applicants assuming they had a prior appointment on a NIH funded T32 training grant. This particular post included some demographic analysis which addressed URM vresus majoritarian status. Leaving aside for a moment the obvious concerns about who gets selected for T32 appointments and where and with whom they are correspondingly training, I was curious about the relative advantage.

Now, the first thing you will recognize is that the critical data are collapsed across all URM groups, although this post doesn’t seem to specify if Asian is included in non-URM, the total N in Table 1 and Table 3 suggest this is the case. This is a little disappointing of course, since the original Ginther report found such striking differences in award rates across URM groups. There was an Asian PI disparity and a Hispanic PI lackthereof, so would it not be super important to keep these groups separate for this post-T32 analysis? Of course it would.

But what got me scratching my head was the presentation of the percentages in Table 3. The show, for example, that 12.9% of non-URM T32 trainees eventually became PI on a funded R01 whereas only 9.8% of URM trainees became PI on a funded R01. That’s all well and good but it obscures the success rate because it doesn’t account for the differential submission rate (24.5% of nonURM T32 trainees eventually submitted an R01, 21.8% of URM trainees).

Doing a little math here, I make it out to be a 52.9% award rate for non-URM applicants and a 45.1% rate for URM applicants. This is a 7.8 percentage point differential which means the URM applicants have a success rate that is 85.3% of the non-URM applicant success rate.

Now we can put this in familiar terms. The Hoppe report found that Black applicants had a success rate that was 60.5% of that enjoyed by white applicants (it was 60.4 in Ginther et al, 2011). So if we take this 24.8 percentage point differential and divide it by the 39.5 percentage point deficit for Black applicants in the Hoppe sample…we end up with about 63% of the Black/white gap reported in Hoppe. The combination of T32 training history, combining of all URM together and adding Asian PIs to the non-URM group closes 63% of the gap.

See my problem here? We want all this broken down so we can answer the most simple question, does T32 training close the gap between Black and white PIs and if so to what extent.

THEN we can go on to ask about other URM populations and the effect of adding the Asian* T32 participants to the non-URM pile.

*reminder that Ginther found that applications with Asian PIs with domestic doctorates had success rates similar to those with white PIs. There is a major complication here with T32 eligibility for Asian-Americans versus Asians who were not T32 eligible as postdocs.

Joe Biden, Hunter Biden, Stimulant Use Disorders and ARPA-H

There was a press release issued by NIDA today, trumpeting a report of a Phase III trial of medication for methamphetamine use disorders. As a very brief recap, we have no approved medications to assist with methamphetamine (or any psychostimulant) use disorder and the distressing health consequences for addiction to cocaine, methamphetamine and some other related drugs are very severe. According to 2019 data from SAMHSA (Table 5.14A), some 534,000 folks over age 25 were in treatment for cocaine use disorder, 566,000 for methamphetamine and this contrasts with 594,000 for heroin, 552,000 for marijuana and 1,957,000 for alcohol. Table 5.1A points out that some 756,000 of folks in this range probably had a cocaine use disorder, 904,000 had a methamphetamine use disorder and 10,959,000 had an alcohol use disorder.

Not everyone who needs treatment gets it. Not even close.

Hunter Biden, the President’s second (and still living) son has recently published a memoir detailing his struggles with addiction to cocaine and alcohol. Joe Biden, the President, issued a recent press release calling for a $6.5 billion launch of an Advanced Research Projects Agency for Health (ARPA-H) within the National Institutes of Health.

With an initial focus on cancer and other diseases such as diabetes and Alzheimer’s, this major investment in federal research and development will drive transformational innovation in health research and speed application and implementation of health breakthroughs.

Notably missing is any prominent mention of substance use disorder in ARPA-H.

On the relative scale of progress in treating cancer and diabetes, and yes even Alzheimer’s, I would argue that treatments for substance use disorders have been woefully under researched. Funding has lagged for the development of treatments and medications both in the public and private sectors. This means novel discovery, of course, but the real glaring deficit is in the routine churning of clinical trial after clinical trial for evaluating pretty run of the mill stuff. As they did in this recent Phase III trial.

Methamphetamine, as they say, is a helluva drug. From Brecht and Herbeck, 2014, we see the following relapse survival curve for a sample of methamphetamine users admitted to the Los Angeles County substance use treatment system. the followup period ranged from 22-90 months over which 23% maintained abstinence from methamphetamine. That means 77% relapsed, with a range of 0-79 months until relapse. As you can see from the below, 36% returned to methamphetamine use immediately upon discharge (Nb, this is not a sample selected for desire to quit), 14% more relapsed by 6 months and a total of 61% had relapsed within a year of entry. The good news, if there is any, is that this should be low hanging fruit. Anything, anything at all, that seems to work will be a huge gain versus the situation at present.

The new trial conducted by Trivedi et al. found that depot injection of naltrexone combined with daily oral buproprion (a cathinone-derivative, aka “bathsalt”) was effective, versus placebo control, in treating methamphetamine use disorder.

“Effective”.

Meaning that within a population of methamphetamine users with “moderate to severe” use disorder who intended to quit, 11.4% responded. Where a response was 3 out of four urine samples negative for methamphetamine during weeks 11-12. Only 1.8% in the placebo group had this “response”. Let’s round that out to 10% efficacy.

Now, the glass is most emphatically half full. Ten percent is not very impressive sounding but it is something. It is some improvement for some folks. Ten percent of the ~904,000 estimated with a methamphetamine use disorder is a lot of people and their families that have improved lives. We are moved by the stories of single individuals- like Hunter Biden, and Nic Sheff and William C. Moyers. Let us apply that same empathy we feel for these men and their relative success at recover to each and every other person with a stimulant use disorder.

And we have nowhere to go but up, with discovery of any additional strategies that, btw, likely will also help with cocaine use disorder.

Do we need DARPA-like innovation? of course. Anti-drug vaccines (something I’ve worked on, for disclosure) have been languishing in a twilight of basic biological efficacy but need a big kick in the pants to advance to real-world efficacy. Wearable technology has several immediately imaginable future uses. Deep brain stimulation. TMS. Individualized therapy based on genotyping. There is no reason to think that we could not go big with ARPA-H for substance use.

It is more than a little bothersome that Joe Biden, who so explicitly ties his interest in Cancer Moonshots and the like to the fate of his older son, does not exhibit the same motivations for the trials of his younger son. Who, btw, is not dead and is at continual risk of relapse given his history.

Trivedi MH, et al. Trial of Bupropion and Naltrexone in Methamphetamine Use Disorder. New England Journal of Medicine. January 14, 2020.

Thoughts on the NIH policy on SABV

There is a new review by Shansky and Murphy out this month which addresses the NIH policy on considering sex as a biological variable (SABV).

Shansky, RM and Murphy, AZ. Considering sex as a biological variable will require a global shift in science culture. Nat Neurosci, 2021 Mar 1. doi: 10.1038/s41593-021-00806-8. Online ahead of print.

To get this out of the way, score me as one who is generally on board with the sentiments behind SABV and one who started trying to change my own approach to my research when this first started being discussed. I even started trying to address this in my grant proposals several cycles before it became obligatory. I have now, as it happens, published papers involving both male and female subjects and continue to do so. We currently have experiments being conducted that involve both male and female subjects and my plan is to continue to do so. Also, I have had many exchanges with Dr. Shansky over the years about these issues and have learned much from her views and suggestions. This post is going to address where I object to things in this new review,for the most part, so I thought I should make these declarations, for what they are worth.

In Box 1, the review addresses a scientist who claims that s/he will first do the work in males and then followup in females as follows:

“We started this work in males, so it makes sense to keep going in males. We will follow up with females when this project is finished.” Be honest, when is a project ever truly finished? There is always another level of ‘mechanistic insight’ one can claim to need. Playing catch-up can be daunting, but it is better to do as much work in both sexes at the same
time, rather than a streamlined follow-up study in females years after the original male work was published. This latter approach risks framing the female work as a lower-impact ‘replication study’ instead of equally valuable to scientific knowledge.

This then dovetails with a comment in Box 2 about the proper way to conduct our research going forward:

At the bare minimum, adhering to SABV means using experimental cohorts that include both males and females in every experiment, without necessarily analyzing data by sex.

I still can’t get past this. I understand that this is the place that the NIH policy on SABV has landed. I do. We should run 50/50 cohorts for every study, as Shansky and Murphy are suggesting here. I cannot for the life of me see the logic in this. I can’t. In my work, behavioral work with rats for the most part, there is so much variability that I am loathe to even run half-size pilot studies. In a lot of the work that I do, N=8 is a pretty good starting size for the minimal ability to conclude much of anything. N=4? tough, especially as a starting size of the groups.

The piece eventually gets around to the notion of how we enforce the NIH SABV policy. As I have pointed out before and as is a central component of this review, we are moving rapidly into a time when the laboratories who claim NIH support for their studies are referencing grant proposals that were submitted under SABV rules.

NOT-OD-15-102 appeared in June of 2015 and warned that SABV policy would “will take effect for applications submitted for the January 25, 2016, due date, and thereafter“. Which means grants to be reviewed in summer 2016, considered at Council in the Fall rounds and potentially funded Dec 1, 2016. This means, with the usual problems with Dec 1 funding dates, that we are finishing up year 4 of some of these initial awards.

One of the main things that Shansky and Murphy address is in the section “Moving forward-who is responsible?“.

whether they have [addressed SABV] in their actual research remains to be seen. NIH grants are nonbinding, meaning that awardees are not required to conduct the exact experiments they propose. Moreover, there is no explicit language from NIH stating that SABV adherence will be enforced once the funds are awarded. Without accountability measures in place, no one is prevented from exclusively using male subjects in research funded under SABV policies.

Right? It is a central issue if we wish to budge the needle on considering sex as a biological variable. And the primary mechanism of enforcement is, well, us. The peers who are reviewing the subsequent grant applications from investigators who have been funded in the SABV era. The authors sortof mention this: “Researchers should be held accountable by making documentation of SABV compliance mandatory in yearly progress reports and by using compliance as a contingency for grant renewals (both noncompetitive and competitive).” Actually, the way this is structured, combined with the following sentence about manuscript review, almost sidesteps the critical issue. I will not sidestep in this post.

We, peer scientists who are reviewing the grant proposals, are the ones who must take primary responsibility to assess whether a PI and associated Investigators have made a good faith attempt to follow/adopt SABV policy or not. Leaving this in the hands of Program to sort out, based on tepid review comments, is a dereliction of duty and will result in frustrating variablity of review that we all hate. So….we are the ones who will either let PIs off the hook, thereby undercutting everything NIH has tried to accomplish, or we will assist NIH by awarding poor scores to applications with a team that has not demonstrably taken SABV seriously. We are at a critical and tenuous point. Will PIs believe that their grants will still be funded with a carefully crafted SABV statement, regardless of whether they have followed through? Or will PIs believe that their grant getting is in serious jeopardy if they do not take the spirit of the SABV policy to heart? The only way this is decided is if the peer review scores reward those who take it seriously and punish those who do not.

So now we are back to the main point of this post which is how we are to assess good-faith efforts. I absolutely agree with Shansky and Murphy that an application (competing or not) that basically says “we’re going to follow up in the females later“, where later means “Oh we didn’t do it yet, but we pinky swear we will do it in this next interval of funding” should not be let off the hook.

However. What about a strategy that falls short of the “bare minimum”, as the authors insist on in Box 2, of including males and females in 50/50 proportion in every experiment, not powered to really confirm any potential sex difference?

I believe we need a little more flexibility in our consideration of whether the research of the PI is making a good faith effort or not. What I would like to see is simply that male and female studies are conducted within the same general research program. Sure, it can be the 50/50 group design. But it can also be that sometimes experiments are in males, sometimes in females. Particularly if there is no particular sense that one sex is always run first and the other is trivially “checked, or that one sex dominates the experimental rationale. Pubs might include both sexes within one paper, that’s the easiest call, but they might also appear as two separate publications. I think this can often be the right approach, personally.

This will require additional advocacy, thinking, pushback, etc, on one of the fundamental principles that many investigators have struggled with in the SABV era. As is detailed in Box 1 and 2 of the review, SABV does not mean that each study is a direct study of sex differences nor that every study in female mammals becomes a study of estrous cycle / ovarian hormones. My experience, as both an applicant and a reviewer, is that NIH study section members often have trouble with this notion. There has not been, in my experience on panels, a loud and general chorus rebutting any such notions during discussion either, we have much ground still to cover.

So we will definitely have to achieve greater agreement on what represents a good faith effort on SABV, I would argue, if we are to advocate strongly for NIH study sections to police SABV with the firm hand that it will require.

I object to what might be an obvious take-away from Shansky and Murphy, i.e., that the 50/50 sample approach is the essential minimum. I believe that other strategies and approaches to SABV can be done which both involve full single-sex sample sizes and do not require every study to be a direct contrast of the sexes in an experimentally clean manner.

A lesson for DEI strategies from the NIH ESI policy

The Director of the NIH, in the wake of a presentation to the Advisory Committee to the Director meeting, has issued a statement of NIH’s commitment to dismantle structural racism.

Toward that end, NIH has launched an effort to end structural racism and racial inequities in biomedical research through a new initiative called UNITE, which has already begun to identify short-term and long-term actions. The UNITE initiative’s efforts are being informed by five committees with experts across all 27 NIH institutes and centers who are passionate about racial diversity, equity, and inclusion. NIH also is seeking advice and guidance from outside of the agency through the Advisory Committee to the Director (ACD), informed by the ACD Working Group on Diversity, and through a Request for Information (RFI) issued today seeking input from the public and stakeholder organizations. The RFI is open through April 9, 2021, and responses to the RFI will be made publicly available. You can learn more about NIH’s efforts, actions, policies, and procedures via a newly launched NIH webpage on Ending Structural Racism aimed at increasing our transparency on this important issue.

This is very much welcome, coming along as it does, a decade after Ginther and colleagues showed that Black PIs faced a huge disadvantage in getting their NIH grants funded. R01 applications with Black PIs were funded at only 58% of the rate that applications with white PIs were funded.

Many people in the intervening years, accelerated after the publication of Hoppe et al 2019 and even further in the wake of the murder of George Floyd at the hands of the Minneapolis police in 2020, have wondered why the NIH does not simply adopt the same solution they applied to the ESI problem. In 2006/2007 the then-Director of NIH, Elias Zerhouni, dictated that the NIH would practice affirmative action to fund the grants of Early Stage Investigators. As detailed in Science by Jocelyn Kaiser

Instead of relying solely on peer review to apportion grants, [Zerhouni] set a floor—a numerical quota—for the number of awards made to new investigators in 2007 and 2008.

A quota. The Big Bad word of anti-equity warriors since forever. Gawd forbid we should use quotas. And in case that wasn’t clear enough

The notice states that NIH “intends to support new investigators at success rates comparable to those for established investigators submitting new applications.” In 2009, that will mean at least 1650 awards to new investigators for R01s, NIH’s most common research grant.

As we saw from Hoppe et al, the NIH funded 256 R01s with Black PIs in the interval from 2011-2015, or 51 per year. In a prior blog post I detailed how some 119 awards to poorer-scoring applications with white PIs could have been devoted to better-scoring proposals with Black PIs. I also mentioned how doing so would have moved the success rate for applications with Black PIs fro 10.7% to 15.6% whereas the white PI success rate would decline from 17.7% to 17.56%. Even funding every single discussed application with a Black PI (44% of the submissions) by subtracting those 1057 applications from the pool awarded with white PIs would reduce the latter applications’ hit rate only to 16.7% which is still a 56% higher rate than the 10.7% rate that the applications with Black PIs actually experienced.

I have been, and continue to be, an advocate for stop-gap measures that immediately redress the funding rate disparity by mandating at least equivalent success rates, just as Zerhouni mandated for ESI proposals. But we need to draw a key lesson from that episode. As detailed in the Science piece

Some program directors grumbled at first, NIH officials say, but came on board when NIH noticed a change in behavior by peer reviewers. Told about the quotas, study sections began “punishing the young investigators with bad scores,” says Zerhouni. That is, a previous slight gap in review scores for new grant applications from first-time and seasoned investigators widened in 2007 and 2008, [then NIGMS Director Jeremy] Berg says. It revealed a bias against new investigators, Zerhouni says.

I don’t know for sure that this continued, but the FY 2012 funding data published by Kienholz and Berg certainly suggest that several NIH ICs continued to fund ESI applications at much lower priority scores/percentiles than were generally required for non-ESI applications to receive awards. And if you examine those NIH ICs pages that publish their funding strategy each year [see the writedit blog for current policy links], you will see that they continue to use a lower payline for ESIs. So. From 2007 to 2021 that is a long interval of policy which is not “affirmative action”, just, in the words of Zerhouni, “leveling the playing field”.

The important point here is that the NIH has never done anything to get to the “real reason” for the fact that early stage investigators proposals were being scored by peer review at lower priority than they, NIH, desired. They have not undergone spasms of reviewer “implicit bias” training. They have not masked the identity of the PI or done anything suggesting they think they can “fix” the review outcome for ESI PIs.

They have accepted the fact that they just need to counter the bias.

NIH will likewise need to accept that they will need to fund Black PI applications with a different payline for a very long time. They need to accept that study sections will “punish*” those applications with even worse scores. They will even punish those applications with higher ND rates. And to some extent, merely by talking about it this horse has left the stall and cannot be easily recalled. We exist in a world where, despite all evidence, white men regularly assert with great confidence that women or minority individuals have all the advantages in hiring and career success.

So all of this entirely predictable behavior needs to be accounted for, expected and tolerated.

__

*I don’t happen to see it as “punishing” ESI apps even further than whatever the base rate is. I think reviewers are very sensitive to perceived funding lines and to reviewing grants from a sort of binary “fund/don’t fund” mindset. Broadcasting a relaxed payline for ESI applications almost inevitably led to reviewers moving their perceived payline for those applications.

Rule Followers

As always Dear Reader, I start with personal confession so you know how to read my biases appropriately.

I am a life time Rule Follower.

I am also a life time Self-Appointed Punisher of Those Who Think the Rules Do Not Apply to Them.

What does “Rule Follower” mean to me? No, not some sort of retentive allegiance to any possible guideline or rule, explicit or implicit. I’ve been known to speed once in awhile. It doesn’t even mean that rule followers are going to agree with, and follow, every rule imaginable for any scenario. It is just an orientation of a person that believes there are such things as rules of behavior, these rules are good things as a social or community compact and that it is a good idea to adhere to them as a general rule. It is a good idea to work within the rules and that this is what is best for society, but also for the self.

The other kind of person, the “Rules Don’t Apply to ME” type, is not necessarily a complete sociopath*. And, in fact, such people may actually be a Rule Follower when it comes to the really big, obvious and Important (in their view) rules. But these are people that do not agree that all of the implicit social rules that Rule Followers follow actually exist. They do not believe that these rules apply to them, and often extend that to the misdemeanor sort of actual formal Rules, aka The Law.

Let’s talk rules of the road- these are the people who routinely speed, California Stop right on reds, and arc into the far lane when making a turn on a multi-lane road. These are the people that bypass a line of patiently waiting traffic and then expect to squeeze into the front of the line with an airy “oops, my badeee, thanks!” smile and wave. They are the ones that cause all sorts of merging havoc because they can’t be arsed to simply go down to the next street or exit to recover from their failure to plan ahead. These are often the people who, despite living in a State with very well defined rules of the road for bicycle traffic, self-righteously violate those rules as a car driver and complain about how the law-compliant bicycle rider is the one in the wrong.

But above all else, these people feel entitled to their behavior. It is an EXTREME OUTRAGE whenever they are disciplined in any way for their selfish and rude behavior that is designed to advantage themselves at the cost to (many) others.

If you don’t let them in the traffic line, you are the asshole. When you make the left turn into lane 2 and they barely manage to keep from hitting you as they fail to arc their own turn properly..you are the asshole. When they walk at you three abreast on the sidewalk and you eyeball the muppethugger trying to edge you off your single lane coming the other way and give every indication you are willing to bodycheck their selfish ass until they finally grudgingly rack it the fuck in…YOU are the asshole.

When they finally get a minor traffic citation for their speeding or failing to stop on a right on red… Oh, sister. It’s forty minutes of complaining rationalization about how unfair this is and why are those cops not solving real crimes and oh woe is me for a ticket they can easily pay. Back in the day when it was still illegal, this was the person caught for a minor weed possession citation who didn’t just pay it but had to go on at length about how outrageous it was to get penalized for their obvious violation of the rules. Don’t even get me started about how these people react to a citation for riding their bicycle on the sidewalk (illegal!) instead of in the street (the law they personally disagree with).

Back before Covid you could identify these two types by hanging around the bulk food bin at your local hippy grocery store. Rule Followers do not sample the items before paying and exiting the store. Those other people…..

Hopefully I’ve chosen examples that get you into the proper mindset of a complex interplay of formal rules that not everyone follows and informal rules of conduct that not everyone follows. I shouldn’t have to draw your attention to how the “Rules Don’t Apply to Me” sail along with convenient interpretations, feigned ignorances and post-hoc everyone-does-it rationales to make their lives a lot easier. That’s right, it’s convenient to not follow the rules, it gets them ahead and frankly those Rule Followers are beta luser cucks for not living a life of personal freedom!

We’re actually in the midst of one of these scenarios right now.

Covid vaccination

As you are aware, there are formal “tiers” being promulgated for who gets schedule for vaccines at which particular time. You know the age cutoffs- we started with 75+ and are now at 65+ in most locations. Then there are the job categories. Health care workers are up first, and then we are working a cascade of importance given occupation. Well, in my environment we had a moment in which “lab workers” were greenlit and Oh, the science lab types rushed to make their appointments. After a short interval, the hammer came down because “lab” meant “lab actually dealing with clinical care and health assessment samples” and not just “any goofaloon who says they work in a lab”.

Trust me, those at the head of that rush (or those pushing as the lab head or institution head) were not the Rule Followers. It was, rather, those types of people who are keen to conveniently define some situation to their own advantage and never consider for a second if they are breaking the Rules.

Then there have been some vaccine situations that are even murkier. We’ve seen on biomedical science tweeter that many lab head prof types have had the opportunity to get vaccinated out of their apparent tier. It seemed, especially in the earlier days prior to vaccine super centers, that a University associated health system would reach the end of their scheduled patients for the day and have extra vaccine.

[ In case anyone has been hiding under a rock, the first vaccines are fragile. They have to be frozen for storage in many cases and thus thawed out. They may not be stable overnight once the vial in question has been opened. In some cases the stored version may need to be “made up” with vehicles or adjuvants or whatever additional components. ]

“Extra” vaccine in the sense of active doses that would otherwise be lost / disposed of if there was no arm to stick it in. Employees who are on campus or close by, can readily be rounded up on short notice, and have no reason to complain if they can’t get vaccinated that particular day, make up this population of arms.

Some Rule Followers were uncomfortable with this.

You will recognize those other types. They were the ones triumphantly posting their good luck on the internet.

In my region, we next started to have vaccine “super centers”. These centers recruited lay volunteers to help out, keep an eye on patients, assist with traffic flow, run to the gloves/syringe depot, etc. And, as with the original health center scenario, there were excess doses available at the end of the day which were offered to the volunteers.

Again, some Rule Followers were uncomfortable with this. Especially because in the early days it was totally on the DL. The charge nurse closest to you would pull a volunteer aside and quietly suggest waiting around at the end of the day just “in case”. It was all pretty sketchy sounding….. to a Rule Follower. The other type of person? NO PROBLEM! They were right there on day one, baby! Vacc’d!

Eventually the volunteer offer policy became someone formalized in my location. Let me tell you, this was a slight relief to a Rule Follower. It for sure decreases the discomfort over admitting one’s good fortune on the intertoobs.

But! It’s not over yet! I mean, these are not formalized processes and the whole vaccine super-center is already chaos just running the patients through. So again, the Rules Don’t Need To Be Followed types are most likely to do the self-advocacy necessary to get that shot in their arm as quickly and assuredly as possible. Remember, it’s only the excess doses that might be available. And you have to keep your head up on what the (rapidly shifting and evolving) procedure might be at your location if you want to be offered vaccine.

Fam, I’m not going to lie. I leaned in hard on anyone I think of as a Rule Follower when I was relating the advantages of volunteering** at one of our vaccine super-centers. I know what we are like. I tell them as much about the chaotic process as I know so as to prepare them for self-advocacy, instead of their native reticence to act without clear understanding of rules that entitle them to get stuck with mRNA.

Still with me?

NIH has been cracking down on URLs in grant applications lately. I don’t know why and maybe it has to do with their recent hoopla about “integrity of review” and people supposedly sharing review materials with outside parties (in clear violation of the review confidentiality RULES, I will note). Anyway, ever since forever you are not supposed to put URL links in your grant applications and reviewers are exhorted never ever to click on a link in a grant. It’s always been explained to me in the context of IP address tracking and identifying the specific reviewers on a panel that might be assigned to a particular application. Whatever. It always seemed a little paranoid to me. But the Rules were exceptionally clear. This was even reinforced with the new Biosketch format that motivated some sort of easy link to one’s fuller set of publications. NIH permits PubMed links and even invented up this whole MyBibliography dealio at MyNCBI to serve this purpose.

Anyway there has been a few kerfuffles of EXTREME ANGER on Science Twitter from applicants who had their proposals rejected prior to review for including URLs. It is an OUTRAGE, you see, that they should be busted for this clear violation of the rules. Which allegedly, according to Those To Whom Rules Do Not Apply, were incredibly arcane rules that they could not possibly be expected to know and waaah, the last three proposals had the same link and weren’t rejected and it isn’t FAAAAAIIIIR!

My gut reaction is really no different than the one I have turning left in a two lane turn or walking at sidewalk hogs. Or the one I have when a habitual traffic law violator finally has to pay a minor fine. Ya fucked around and found out. As the kids say these days.

For some additional perspective, I’ve been reviewing NIH grants since the days when paper hard copies were submitted by the applicant and delivered to the reviewers as such. Pages could be missing if the copier effed up- there was no opportunity to fix this once a reviewer noticed it one week prior to the meeting. Font size shenanigans were seemingly more readily played. And even in the days since, as we’ve moved to electronic documents, there are oodles and oodles of rules for constructing the application. No “in prep” citations in the old Biosketch….people did it anyway. No substituting key methods in the Vertebrate Animals section…..people still do it anyway. Fonts and font size, okay, but what about vertical line spacing….people fudge that anyway. Expand figure “legends” (where font size can be smaller) to incorporate stuff that (maybe?) should really be in the font-controlled parts of the text. Etc, etc, etc.

And I am here to tell you that in many of these cases there was no formal enforcement mechanism. Ask the SRO about a flagrant violation and you’d get some sort of pablum about “well, you are not obliged to consider that material..”. Font size? “well…..I guess that’s up to the panel”. Which is enraging to a Rule Follower. Because even if you want to enforce the rules, how do you do it? How do you “ignore” that manuscript described as in prep, or make sure the other reviewers do? How do you fight with other reviewers about how key methods are “missing” when they are free to give good scores even if that material didn’t appear anywhere in figure legend, Vertebrate Animals or, ISYN, a 25% of the page “footnote” in microfont. Or how do your respond if they say “well, I’m confident this investigator can work it out”?

If, in the old days, you gave a crappy score to a proposal that everyone loved by saying “I put a ruler on the vertical and they’ve cheated” the panel would side eye you, vote a fundable score and fuck over any of your subsequent proposals that they read.

Or such might be your concern if your instinct was to Enforce the Rules.

Anyway, I’m happy to see CSR Receipt and Referral enforce rules of the road. I don’t think it an outrage at all. The greater outrage is all the people who have been able to skirt or ignore the rules and advantage themselves against those of us who do follow the rules***.

__

*Some of my best friends are habitual non-followers-of-rules.

**I recommend volunteering at a vaccine super station if you have the opportunity. It is pretty cool just to see how your health care community is reacting in this highly unusual once-in-a-generation crisis. And its cool, for those of us with zero relevant skills, to have at least a tiny chance to help out. Those are the Rules, you know? 🙂

***Cue Non-Followers-of-Rules who, Trumplipublican- and bothsiders-media-like, are absolutely insistent then when they manage to catch a habitual Rule Follower in some violation it proves that we’re all the same. That their flagrant and continual behavior is somehow balanced by one transgression of someone else.

Merit of NIH Grants Depends on the Reviewer’s Opinion

This is not news to this audience but it bears addressing in as many ways as possible, in the context of the Hoppe et al 2019 and Ginther et al 2011 findings. Behind most of the resistance to doing anything about the funding disparity for investigators and, as we’re now finding out, topics is still some lingering idea that the NIH grant selection process is mostly about merit.

Objective merit. Sure we sort of nod that we understand that there is some wiggle room but overall it is difficult to find anyone that appears to understand something in a deep way.

“Merit” of NIH grants is untethered to anything objective. It relies on the opinion of the peer reviewers. The ~3 reviewers who are assigned to do deep review and the members of the panel (which can be 20-30ish folks) who vote scores after the discussion.

For NIH grant review-experienced tweeps, do you ever experience a panel voting a fundable score for an application you personally think does not deserve funding?

— Drug Monkey (@drugmonkeyblog) January 27, 2021

This particular dumb twitter poll shows that 77% of experienced reviewers either occasionally or regularly have the experience of thinking a grant that should not receive funding is very likely to do so.

For NIH grant review-experienced tweeps, do you ever experience a panel voting a clearly not-fundable score for an application you personally think quite obviously deserves funding?

— Drug Monkey (@drugmonkeyblog) January 27, 2021

and this other dumb little twitter poll shows that 88% of experienced NIH grant reviewers either occasionally or frequently experience a panel voting a non-fundable score for a grant they think deserves funding.

It will not escape you that individual reviewers tend to think a lot more grants should be funded than can be funded. And this shows up in the polls to some extent.

This is not high falutin’ science and it is possible we have some joker contamination here from people who are not in fact NIH review experienced.

But with that caveat, it tends to support the idea that the mere chance of which individuals are assigned to review a grant can have a major effect on merit. After all, the post-discussion scores of these individuals tends to significantly constrain the voting. But the voting is important too, since panel members can go outside the range or decide en masse to side with one or the other ends of the post-discussion range.

Swap out the assigned reviewers for a different set of three individuals and the outcomes are likely to be very different. Swap out one panel for another and the tendencies could be totally different. Is your panel heavy in those interested in sex differences and/or folks heavily on board with SABV? Or is it dominated by SABV resisters?

Is the panel super interested in the health effects of cannabis and couldn’t give a fig about methamphetamine? What do YOU think is going to come out of that panel with fundable scores?

Does the panel think any non-mammalian species is horrible for modeling human health and should really never be funded? Does the panel geek away at tractable systems and adore anything fly or worm driven and complain about the lack of manipulability available in a rat?

Of course you know this. These kinds of whines and complaints are endemic to fireside chats whenever two or more NIH grant-seeking investigators are present!

But somehow when it is a disparity of race or of topics of interest to minority communities in the US, such as from Hoppe et al 2019, then nobody is concerned. Even when there are actual data on the table showing a funding disparity. And everyone asks their “yeahbutwhatabout” questions, springing right back into the mindset that at the very core the review and selection of grants is about merit. The fact their worm grant didn’t get selected is clear evidence of a terrible bias in the NIH approach. The fact African-American PIs face a payline far lower than they do…..snore.

Because in that case it is about objective merit.

And not about the coincidence of whomever the SRO has decided should review that grant.

Susan Rice Empowered to Hold NIH Director Collins to Account on the Funding Discrimination

On his first day in office President Biden signed an Executive Order described thusly by the NYT. I am having difficulty finding a link to the exact text right now. [Edited to Add: The Executive Order On Advancing Racial Equity and Support for Underserved Communities Through the Federal Government.]

“The president designated Susan E. Rice, who is the head of his Domestic Policy Council, as the leader of a “robust, interagency” effort requiring all federal agencies to make “rooting out systemic racism” central to their work. His order directs the agencies to review and report on equity in their ranks within 200 days, including a plan on how to remove barriers to opportunities in policies and programs. The order also moves to ensure that Americans of all backgrounds have equal access to federal government resources, benefits and services. It starts a data working group as well as the study of new methods to measure and assess federal equity and diversity efforts.”

Well, the NIH doesn’t have to take 200 days to “review and report”. They’ve already done so in

Ginther, D.K., Schaffer, W.T., Schnell, J., Masimore, B., Liu, F., Haak, L.L., Kington, R., 2011. Race, ethnicity, and NIH research awards. Science 333(6045), 1015-1019.

and

Hoppe, T.A., Litovitz, A., Willis, K.A., Meseroll, R.A., Perkins, M.J., Hutchins, B.I., Davis, A.F., Lauer, M.S., Valantine, H.A., Anderson, J.M., Santangelo, G.M., 2019. Topic choice contributes to the lower rate of NIH awards to African-American/black scientists. Sci Adv 5(10), eaaw7238.

So we can skip the 200 days worth of can kicking, Dr. Collins, and move straight to the fixing part. The “ensure” part. The “equal access” part.

This means funding research on topics that are important to Americans of all backgrounds, including African-American ones. Equally. This means pumping up the budget of the National Institute on Minority Health and Health Disparities (NIMHD). It also means holding the other ICs responsible for taking on their share of these projects and not just shrugging them off into NIMHD.

It means funding not just white American science teams that work on these topics but funding teams of African-American investigators. Equally.

It also means not just funding African-American professors to work on topics of relevance to the health interests of African-Americans but rather equalizing the funding chances of African-American PIs who choose to work on any topic at all.

It’s time to go big. Forces within the NIH who have been trying to do good on this should feel empowered to shout down the nay sayers and to hold the foot draggers to account.

Forces outside the NIH who have been trying to do good on this should likewise feel empowered to hold Susan Rice, their Senators and Congress Reps to account.

NIH Awards for ESI with preliminary data verboten!

The NIH has launched a new FOA called the Stephen I. Katz Early Stage Investigator Research Project Grant (Open Mike blog post). PAR-21-038 is the one for pre-clinical, PAR-21-039 is the one for clinical work. These are for Early Stage Investigators only and have special receipt dates (e.g. January 26, 2021; May 26, 2021; September 28, 2021). Details appear to be a normal R01- up to 5 years and any budget you want to try (of course over $500k per year requires permission).

The novelty here appears to be entirely this:

For this FOA, applications including preliminary data will be considered noncompliant with the FOA instructions and will be withdrawn. Preliminary data are defined as data not yet published. Existence of preliminary data is an indication that the proposed project has advanced beyond the scope defined by this program and makes the application unsuitable for this funding opportunity. Publication in the proposed new research direction is an indication that the proposed work may not be in a new research direction for the ESI.

This will be fascinating. A little bit more specification that the scientific justification has to rest on published (or pre-printed) work only:

The logical basis and premise for the proposed work should be supported by published data or data from preprints that have a Digital Object Identifier (DOI). These data must be labeled and cited adjacent to each occurrence within the application and must be presented unmodified from the original published format. Figures and tables containing data must include citation(s) within the legend. The data should be unambiguously identified as published through citation that includes the DOI (see Section IV.2). References and data that do not have an associated DOI are not allowed in any section of the application. Prospective applicants are reminded that NIH instructions do not allow URLs or hyperlinks to websites or documents that contain data in any part of the application

So how is this going to work in practice for the intrepid ESI looking to apply for this?

First, there is no reason you have to put the preliminary data you have available in the application. One very hot comment over at the Open Mike blog post about the proposals being unsupported and therefore the projects will be doomed to failure is totally missing this point. PIs are not stupid. They aren’t going to throw up stupid ideas, they are going to propose their good ideas that can be portrayed as being unsupported by preliminary data.

Twill be interesting to see how this is interpreted vis a vis meeting presentations, seminars and (hello!) job talks. What is a reviewer expected to do if they see an application without any preliminary data per the FOA, but have just seen a relevant presentation from the applicant which shows that Aim 1 is already completed? Will they wave a flag? See above, the FOA says the “existence” of preliminary data, not the “inclusion” of preliminary data will make the app non-compliant.

But there is an aspect of normal NIH grant review that is not supposed to depend on “secret” knowledge, i.e., that available only to the reviewer, not published. So it is frowned upon for a reviewer to say “well the applicant gave a seminar last month at our department and showed that this thing will work”. It’s special knowledge only available to that particular reviewer on the panel. Unverifiable.

This would be similar, no?

Or is this more like individual knowledge that the PI had faked data? In such cases the reviewers are obligated to report that to the SRO in private but not to bring it up during the review.

If they ARE going to enforce the “existence” of relevant preliminary data, how will it be possible to make this fair? It will be entirely unfair. Some applicants will be unlucky enough to have knowledgeable whistle blowers on the panel and some will evade that fate by chance. Reviewers being what they are, will only variably respond to this charge to enforce the preliminary data thing, even if semi-obligated. After all, what is the threshold for the data being specifically supportive of the proposal at hand?

Strategy-wise, of course I endorse ESI taking advantage of this. The FOAs list almost all of the ICs with relevant funding authority if I counted correctly (fewer for the human-subjects one, of course). There is an offset receipt date, so it keeps the regular submission dates clear. You can put one in, keep working on it and if the prelim data look good, put a revised version in for a regular FOA next time. Or, if you can’t work on it or the data aren’t going well, you can resubmit “For Resubmissions, the committee will evaluate the application as now presented, taking into consideration the responses to comments from the previous scientific review group and changes made to the project.” Win-win.

Second strategy thing. This is a PAR and the intent is to convene panels for this mechanism. This means that your relative ESI advantage at the point of review disappears. You are competing only against other ESI. Now, how each IC chooses to prioritize these is unknown. But once you get a score, you are presumably just within whatever ESI policy a given IC has set for itself.

I’m confused by the comments over at Open Mike. They seem sort of negative about this whole thing. It’s just another FOA, folks. It doesn’t remove opportunities like the R15. No it doesn’t magically fix every woe related to review. It is an interesting attempt to fix what I see as a major flaw in the evolved culture of NIH grant review and award. Personally I’d like to see this expanded to all applicants but this is a good place to start.

Black PIs working on the “right” topics are at a further NIH funding disadvantage

One of the potential takeaway messages from the Hoppe et al 2019 finding, and the Open Mike blogpost, is that if Black PIs want to have a better success rate for their applications, perhaps they should work on the right topics. The “right” topics, meaning the ones that enjoy the highest success rates. After all, the messaging around the release of Hoppe was more or less this: Black PI apps are only discriminated against because they are disproportionately proposed on topics that are discriminated against.

(Never mind that right in the Abstract of Hoppe they admit this only explains some 20% of the funding gap.)

We find, however, a curious counter to this message buried in the Supplement. I mentioned this in a prior post but it bears posting the data for a more memorable impact.

The left side of Figure S6 in Hoppe et al. shows the percent of applications within both the African-American/Black and WHite distributions that were submitted which landed in topic clusters across the success rate quintiles. The 1st is the best, i.e, the most readily funded topic cluster. We can see from this that while applications with white PIs are more or less evenly distributed, the applications with Black PIs are less frequently landing in the best funded topic clusters and more frequently landing in the lowest funded topic clusters. Ok, fine, this is the distributional description that underlies much of the takeaway messaging. On the right side of Figure S6, there is a Table. No idea why they chose that instead of a graph but it has the tendency to obscure a critical point.

Here I have graphed the data, which is the success rate for applications which fall into the topic-success quintiles by the race of the PI. This, first of all, emphasizes the subvocalized admission that even in the least-fundable topic clusters, applications with white PIs enjoyed a success advantage. Actually this main effect was present in each quintile, unsurprisingly. What also immediately pops out, in a way it does not with the table representation, is that in the best funded topic area the advantage of applications with white PIs is the greatest. Another way to represent this is by calculating the degree to which applications with Black PIs are disadvantaged within each quintile.

This represents the success for applications with Black PIs as a percentage of the success for applications with white PIs. As you can see, the biggest hit is at the first and fifth quintiles with the applications faring the best at the middle topic-success quintile. Why? Well one could imagine all kinds of factors having to do with the review of applications in those topic domains. The OpenMike blog post on ICs with lower funding rates (because they have tiny budgets, in part) may explain the fifth quintile but it wouldn’t apply to the top quintile. In fact quite the contrary. Ok, this depiction speaks to the relative hit to success rates within quintile. But the applicant might be a little more interested in the raw percentile hit, especially given the cumulative probability distributions we were discussing yesterday. Recall, the average difference was 7 percentile points (17.7% for Wh PI apps vs 10.7% for Black PI apps).

The disparity is highest in the first quintile. It is a hit of 10.8 percent, as opposed to the all-apps average hit of 7.0 percent.

Obviously we cannot draw much more from the available data. But it certainly cautions us that pushing Black applicants to work on the “right” topics is not a clear solution and may even be counter productive. This is on the acute level of a current PI deciding what to propose in an application, and what to pursue with a multi-application strategy over time. But it also, importantly, serves up a caution for pipeline solutions that try to get more Black trainees into the right labs so that they will work on the right topics using, in Dr. Collins’ parlance, “upgraded” “methodologies”. If this topic/race disparity is not resolved by the time these new trainees hit Assistant Professor stage, we are going to push more Black Professors into research domains that are even harder to succeed in.

We may eventually get some more insight. CSR promised this summer to start looking into study section behavior more closely. It may be that fewer applications from Black PIs in the most successful topic domains is due to disproportionately fewer Black PIs in those fields which leads to fewer of them on the relevant study sections. Even absent that factor, a lower presence in the fields of interest may drive more implicit or explicit bias against the ones that do choose those fields. We just can’t tell without more information about the constitution of study sections and the success rates that emerge from them. Oh, and the exception pay behavior of the Branches and Divisions within each IC. That’s also important to examine as it may relate to topic domain.

Historical success rates predict current attitudes, redux

It is hard to overstate the problem that plummeting success rates at the NIH have caused for biomedical science careers. We have expectations for junior faculty that were developed in the 1980s and maybe into the 90s. Attitudes that are firmly entrenched in our senior faculty who got their first awards in the 1980s or even the 1970s…and then were poised to really rake it in during the doubling interval (since undoubled). Time for a trip down memory lane.

The red trace depicts success rates from 1962 to 2008 for R01 equivalents (R01, R23, R29, R37). These are not broken down by experienced/new investigators status, nor are new applications distinguished from competing continuation applications. The blue line shows total number of applications reviewed and the data in the 60s are listed as “estimated” success rates. (source)

The extension of these data into more recent FY can be found over at the RePORTER. I like to keep my old graph because NIH has this nasty tendency to disappear the good old days so we’ll forget about how bad things really are now. From 2011 to 2017 success rates hovered from 17 to 19% and in the past two years we’ve seen 21-22% success.

In the historical trends from about 1980 to the end of the doubling in 2002 we see that 30% success rates ruled the day as expected average. Deviations were viewed as disaster. In fact the doubling of the NIH budget over a decade was triggered by the success rates falling down into the 25% range and everyone screaming at Congress for help. For what it is worth, the greybeards when I was early career were still complaining about funding rates in the early 1980s. Was it because they were used to the 40% success years right before that dropping down to 30%? Likely. When they were telling us “it’s all cyclical, we’ve seen this before on a decade cycle” during the post-doubling declines….well it was good to see these sorts of data to head off the gaslighting, I can tell you.

Anyway, the point of the day is that folks who had a nice long run of 30% success rates (overall; it was higher once you were established, aka had landed one grant) are the ones who set, and are setting, current expectations. Today’s little exercise in cumulative probability of grant award had me thinking. What does this analysis look like in historical perspective?

I’m using the same 17.7% success rate for applications with white PIs reported in Hoppe et al and 30% as a sort of historical perspective number. Relevant to tenure expectations, we can see that the kids these days have to work harder. Back in the day, applicants had a 83.2% cumulative probability of award with just 5 applications submitted. Seems quaint doesn’t it? Nowadays a white PI would have to submit 9 applications to get to that same chance of funding.

How does that square with usual career advice? Well, of course newbs should not submit R01 in the first year. Get the lab up and running on startup, maybe get a paper, certainly get some solid preliminary data. Put the grant in October in year 2 (triaged), wait past a round to do a serious revision, put it in for July. Triaged again in October of Year 3. Two grants in, starting Year 3. Well now maybe things are clicking a bit so the PI manages to get two new proposals together for Oct and/or Feb and if the early submission gets in, another revision for July. So in Fall of Year 4 we’re looking at four or five submissions with a fairly good amount of effort and urgency. This could easily stretch into late Year 4.

Where do the kids these days fit in four more applications?

The cumulative probability of getting NIH grants

One of the career strategies we have discussed numerous times in various contexts is how many grant applications one should be submitting to the NIH. I have been a consistent advocate for …more. This is a recognition that success rates on a per-application basis have been below 20% for most of my career. Obviously this particular target number varies a lot. Sometimes we are talking about paylines, since that seems to be a hard target for success. Recent paylines from the NCI have been in the high single digits- 7-9%. Or we may talk about NIH-wide success rates overall, accounting for not just payline funding but pickups above the payline. These numbers change from year to year but mid to upper teens is a pretty fair general estimate.

My usual argument is that investigators who want to get funded should start with the assumption that they are no better than anyone else and need to base their strategy off the average success rate….at the very least.

Dumb old me, math challenged in the extreme, may have even expressed this as something like “If the payline is 10%, you need to put in 10 applications to even be in the game”. The more math savvy of you immediately chime in to correct me about calculating cumulative probabilities. This is not a difficult concept. I get it. But my mind likes to forget about it and I’ve never taken a blog whack at this issue directly, that I can recall.

Thanks to this handy binomial probability tool I googled up, we can now contemplate the abyss. Let us suppose a per-application success rate of 17.7%. Mid to upper teens, a decent place to start our discussion. And let us gate on the cumulative probability of at least one award. Well, if you put in 5 applications, your odds of one of them funding is 62.2% and if you put in 10 applications, this is 85.7%. Not too shabby. But it puts a very fine point that probabilities of award do not add. Fifteen applications are required to get to a 95% cumulative probability of at least one being awarded.

Reminder: We are not talking payline here. We are talking the 17.7% hit rate for the NIH wide average of everything that goes into awards including all of those various pickup behaviors. If you want the assurance of making the payline at a place like NCI, well…..Lord help you, amirite? That sounds totally rough and brutal and even unfair.

Now. Suppose that for some reason, say that your skin reflectance categorizes you as Black instead of white, your success rate was 10.7% instead of 17.7%.

Your chances are, of course, somewhat different. The cumulative advantage of putting in more grants, aka working harder, accrues less surely here as well. I’ve color coded a few ~equivalent cumulative probabilities for convenience. Using the Hoppe success rates, a Black applicant has to put in 9 proposals to get approximately the same 62% chance a white applicant achieves with only 5 applications. This same PI would have to put in 18 proposals to approximate the 86% hit rate the white PI gets with only 10 applications. About 25 proposals to get the 95% hit rate enjoyed by white applicants who put in 15 proposals.

Insert my favorite staring eyes emojii available on other platforms.

I would estimate that many Black folks, in the academy and elsewhere, are somewhat used to the idea that they need to grind a bit harder to achieve. At some level it probably doesn’t even really bother some on the day to day.

But this is a LOT harder. Putting in just one grant proposal is not easy. Particularly when you are a brand new Professor. But it is not trivial, even when you are a Full professor with some time and experience under your belt. [ Oh, btw, sidebar: Ginther et al 2018 has a little comment that I probably missed originally. “In results not reported, being a full professor increased the probability of NIH funding by 11.9 ppt (p < .001), but the funding gap remained -12.8 ppt (p < .001).” Yeah, age doesn’t help. ] When we are talking 5, 10, 25 applications it is maybe easy to overlook the sweat that goes into making new and credible proposals. Sure, some can be revised proposals and some are retreads of various prior proposals. But they take work and sweat to make them competitive. You are not going to enjoy the NIH-wide average hit rate with consistently below-average proposals!

This brings me back to a related issue that appeared in the Ginther et al 2011 report. “Applications from black and Asian investigators were significantly less likely to receive R01 funding compared with whites for grants submitted once or twice……black and Asian investigators are less likely to be awarded an R01 on the first or second attempt, blacks and Hispanics are less likely to resubmit a revised application, and black investigators that do resubmit have to do so more often to receive an award.“. I will admit I still don’t quite understand what they are presenting here at the end. It reads as though they are gating on Black investigators who do eventually win an award and do so on revision, not the A0 (this sample was back in the A2-permitted days, iirc). This whole passage, however, can be received as “well, just work a little harder to compensate” and to appear as if we’re only talking about an extra revision or two. I probably received this in this way myself on initially seeing the 2011 paper. And I have to say the “1.7 fold advantage” that is discussed in Hoppe for the per-application success rates comes in the same way. It can be received as, well, you just have to write two to get one. Because it focuses us on the “given you did get an award” instead of what it takes to get that award, statistically speaking.

But looking at these cumulative probability graphs really hits differently.

Black PIs don’t have to work just a little harder.

Black PIs have to work a LOT harder.