Disparity of NSF funding

You are familiar with the #GintherGap, the disparity of grant award at NIH that leaves the applications with Black PIs at substantial disadvantage. Many have said from the start that it is unlikely that this is unique to the NIH and we only await similar analyses to verify that supposition.

Curiously the NSF has not, to my awareness, done any such study and released it for public consumption.

Well, a group of scientists have recently posted a preprint:

^{Chen, C. Y., Kahanamoku, S. S., Tripati, A., Alegado, R. A., Morris, V. R., Andrade, K., & Hosbey, J. (2022, July 1). Decades of systemic racial disparities in funding rates at the National Science Foundation. OSF Preprints. July 1. doi:10.31219/osf.io/xb57u.}

It reviews National Science Foundation awards (from 1996-2019) and uses demographics provided voluntarily by PIs. They found that the applicant PIs were 66% white, 3% Black, 29% Asian and below 1% for each of American Indian/Alaska Native and Native Hawaiian/Pacific Islander groups. They also found that across the reviewed years, the overall funding rate varied from 22%-34%, so the data were represented as the rate for each group relative to the average for each year. In Figure 1, reproduced below, you can see that applications with white PIs enjoy a nice consistent advantage relative to other groups and the applications with Asian PIs suffer a consistant disadvantage. The applications with Black PIs are more variable year over year but are mostly below average except for 5 years when they are right at the average. The authors note this means that in 2019, there were 798 awards with white PIs above expected value, and 460 fewer than expected awarded with Asian PIs. The size of the disparity differs slightly across the directorates of the NSF (there are seven, broken down by discipline such as Biological Sciences, Engineering, Math and Physical Sciences, Education and Human Resources, etc) but the same dis/advantage based on PI race remains.

Fig 1B from Chen et al. 2022 preprint

It gets worse. It turns out that these numbers include both Research and Non-Research (conference, training, equipment, instrumentation, exploratory) awards. Which represent 82% and 18% of awards, with the latter generally being awarded at 1.4-1.9 times the rate for Research awards in a given year. For white

Fig 3 from Chen et al 2022 preprint FY 13 – 19;
open = Non-Research, closed = Research

PI applications the two types both are funded at higher than the average rate, however significant differences emerge for Black and Asian PIs with Research awards having the lower probability of success.

So why is this the case. Well, the white PI applications get better scores from extramural reviewers. Here, I am not expert in how NSF works. A mewling newbie really. But they solicit peer reviewers which assign merit scores from 1 (Poor) to 5 (Excellent). The preprint shows the distributions of scores for FY15 and FY16 Research applications, by PI race, in Figure 5. Unsurprisingly there is a lot of overlap but the average score for white PI apps is superior to that for either Black or Asian PI apps. Interestingly, average scores are worse for Black PI apps than for Asian PI apps. Interesting because the funding disparity is larger for Asian PIs than for Black PIs. And as you can imagine, there is a relationship between score and chances of being funded but it is variable. Kind of like a Programmatic decision on exception pay or the grey zone function in NIH land. Not sure exactly how this matches up over at NSF but the first author of the preprint put me onto a 2015 FY report on the Merit Review Process that addresses this. Page 74 of the PDF (NSB-AO-206-11) has a Figure 3.2 showing the success rates by average review score and PI race. As anticipated, proposals in the 4.75 (score midpoint) bin are funded at rates of 80% or better. About 60% for the 4.25 bin, 30% for the 3.75 bin and under 10% for the 3.25 bin. Interestingly, the success rates for Black PI applications are higher than for white PI applications at the same score. The Asian PI success rates are closer to the white PI success rates but still a little bit higher, at comparable scores. So clearly something is going on with funding decision making at NSF to partially counter the poorer scores, on average, from the reviewers. The Asian PI proposals do not have as much of this advantage. This explains why the overall success rates for Black PI applications are closer to the average compared with the Asian PI apps, despite worse average scores.

Fig 5 from Chen et al 2022 preprint

One more curious factor popped out of this study. The authors, obviously, had to use only the applications for which a PI had specified their race. This was about 96% in 1999-2000 when they were able to include these data. However it was down to 90% in 2009, 86% in 2016 and then took a sharp plunge in successive years to land at 76% in 2019. The first author indicated on Twitter that this was down to 70% in 2020, the largest one year decrement. This is very curious to me. It seems obvious that PIs are doing whatever they think is going to help them get funded. For the percentage to be this large it simply has to involve large numbers of white PIs and likely Asian PIs as well. It cannot simply be Black PIs worried that racial identification will disadvantage them (a reasonable fear, given the NIH data reported in Ginther et al.) I suspect a certain type of white academic who has convinced himself (it’s usually a he) that white men are discriminated against, that the URM PIs have an easy ride to funding and the best thing for them to do is not to declare themselves white. Also another variation on the theme, the “we shouldn’t see color so I won’t give em color” type. It is hard not to note that the US has been having a more intensive discussion about systemic racial discrimination, starting somewhere around 2014 with the shooting of Michael Brown in Ferguson MO. This amped up in 2020 with the strangulation murder of George Floyd in Minneapolis. Somewhere in here, scientists finally started paying attention to the Ginther Gap. News started getting around. I think all of this is probably causally related to sharp decreases in the self-identification of race on NSF applications. Perhaps not for all the same reasons for every person or demographic. But if it is not an artifact of the grant submission system, this is the most obvious conclusion.

There is a ton of additional analysis in the preprint. Go read it. Study. Think about it.

Additional: Ginther et al. (2011) Race, ethnicity, and NIH research awards. Science, 2011 Aug 19; 333(6045):1015-9. [PubMed]

NIH data on Discussion Rates and Grant Submitting Vigor

The latest blog post over at Open Mike, from the NIH honcho of extramural grant award Mike Lauer, addresses “Discussion Rate”. This is, in his formulation, the percent of applicants (in a given Fiscal Year, FY21 in this case) who are PI on at least one application that reaches discussion. I.e., not triaged. The post presents three Tables, with this Discussion rate (and Funding rate) presented by the Sex of the PI, by race (Asian, Black, White only) or ethnicity (Hispanic or Latino vs non-Hispanic only). The tables further presented these breakdowns by Early Stage Investigator, New Investigator, At Risk and Established. At risk is a category of “researchers that received a prior substantial NIH award but, as best we can tell, will have no funding the following fiscal year if they are not successful in securing a competing award this year.” At this point you may wish to revisit an old blog post by DataHound called “Mind the Gap” which addresses the chances of regaining funding once a PI has lost all NIH grants.

I took the liberty of graphing the By-Race/Ethnicity Discussion rates, because I am a visual thinker.

There seem to be two main things that pop out. First, in the ESI category, the Discussion rate for Black PI apps is a lot lower. Which is interesting. The 60% rate for ESI might be a little odd until you remember that the burden of triage may not fall on ESI applications. At least 50% have to be discussed in each study section, small numbers in study section probably mean that on average it is more than half, and this is NIH wide data for FY 21 (5,410 ESI PIs total). Second, the NI category (New, Not Early on the chart) seems to suffer relative to the other categories.

Then I thought a bit about this per-PI Discussion rate being north of 50% for most categories. And that seemed odd to me. Then I looked at another critical column on the tables in the blog post.

The Median number of applications per applicant was…. 1. That means the mode is 1.

Wow. Just….wow.

I can maybe understand this for ESI applicants, since for many of them this will be their first grant ever submitted.

but for “At Risk”? An investigator who has experience as a PI with NIH funding, is about to have no NIH funding if a grant does not hit, and they are submitting ONE grant application per fiscal year?

I am intensely curious how this stat breaks down by deciles. How many at risk PIs are submitting only one grant proposal? Is it only about half? Two-thirds? More?

As you know, my perspective on the NIH grant getting system is that if you have only put in one grant you are not really trying. The associated implication is that any solutions to the various problems that the NIH grant award system might have that are based on someone not getting their grant after only one try are not likely to be that useful.

I just cannot make this make sense to me. Particularly if the NIH

It is slightly concerning that the NIH is now reporting on this category of investigator. Don’t get me wrong. I believe this NIH system should support a greater expectation of approximately continual funding for investigators who are funded PIs. But it absolutely cannot be 100%. What should it be? I don’t know. It’s debatable. Perhaps more importantly who should be saved? Because after all, what is the purpose of NIH reporting on this category if they do not plan to DO SOMETHING about it? By, presumably, using some sort of exception pay or policy to prevent these at risk PIs from going unfunded.

There was just such a plan bruited about for PIs funded with the ESI designation that were unable to renew or get another grant. They called them Early Established Investigators and described their plans to prioritize these apps in NOT-OD-17-101. This was shelved (NOT-OD-18-214) because “NIH’s strategy for achieving these goals has evolved based on on-going work by an Advisory Committee to the Director (ACD) Next Generation Researchers Initiative Working Group and other stakeholder feedback” and yet asserted “NIH..will use an interim strategy to consider “at risk investigators”..in its funding strategies“. In other words, people screamed bloody murder about how it was not fair to only consider “at risk” those who happened demographically to benefit from the ESI policy.

It is unclear how these “consider” decisions have been made in the subsequent interval. In a way, Program has always “considered” at risk investigators, so it is particularly unclear how this language changes anything. In the early days I had been told directly by POs that my pleas for an exception pay were not as important because “we have to take care of our long funded investigators who will otherwise be out of funding”. This sort of thing came up in study section more than once in my hearing, voiced variously as “this is the last chance for this PIs one grant” or even “the PI will be out of funding if…”. As you can imagine, at the time I was new and full of beans and found that objectionable. Now….well, I’d be happy to have those sentiments applied to me.

There is a new version of this “at risk” consideration that is tied to the new PAR-22-181 on promoting diversity. In case you are wondering why this differs from the famously rescinded NINDS NOSI, well, NIH has managed to find themselves a lawyered excuse.

Section 404M of the Public Health Service Act (added by Section 2021 in Title II, Subtitle C, of the 21st Century Cures Act, P.L. 114-255, enacted December 13, 2016), entitled, “Investing in the Next Generation of Researchers,” established the Next Generation Researchers Initiative within the Office of the NIH Director.  This initiative is intended to promote and provide opportunities for new researchers and earlier research independence, and to maintain the careers of at-risk investigators.  In particular, subsection (b) requires the Director to “Develop, modify, or prioritize policies, as needed, within the National Institutes of Health to promote opportunities for new researchers and earlier research independence, such as policies to increase opportunities for new researchers to receive funding, enhance training and mentorship programs for researchers, and enhance workforce diversity;

“enacted December 13, 2016“. So yeah, the NOSI was issued after this and they could very well have used this for cover. The NIH chose not to. Now, the NIH chooses to use this aspect of the appropriations language. And keep in mind that when Congress includes something like this NGRI in the appropriations language, NIH has requested it or accepted it or contributed to exactly how it is construed and written. So this is yet more evidence that their prior stance that the “law” or “Congress” was preventing them from acting to close the Ginther Gap was utter horseshit.

Let’s get back to “at risk” as a more explicitly expressed concern of the NIH. What will these policies mean? Well, we do know that none of this comes with any concrete detail like set aside funds (the PAR is not a PAS) or ESI-style relaxation of paylines. We do know that they do this all the damn time, under the radar. So what gives? Who is being empowered by making this “consideration” of at-risk PI applications more explicit? Who will receive exception pay grants purely because they are at risk? How many? Will it be in accordance with distance from payline? How will these “to enhance diversity” considerations be applied? How will these be balanced against regular old “our long term funded majoritarian investigator is at risk omg” sentiments in the Branches and Divisions?

This is one of the reasons I like the aforementioned Datahound analysis, because at least it gave a baseline of actual data for discussion purposes. A framework a given I or C could follow in starting to make intelligent decisions.

What is the best policy for where, who, what to pick up?

Research colonialism, analyzing an RFA

I recently fielded a question from a more junior scientist about what, I think, has been termed research colonialism with specificity to the NIH funding disparity known as the Ginther Gap. One of the outcomes of the Hoppe et al 2019 paper, and the following Lauer et al 2021, was a call for a hard look at research on the health issues of communities of color. How successful are grant proposals on those topics, which ICs are funding them, what are the success rates and what are the budget levels appropriated to, e.g. the NIMHD. I am very much at sea trying to answer the question I was asked, which boiled down to “Why is it always majoritarian PIs being funded to do research with communities of color?”. I really don’t know how to answer that or how to begin to address it with NIH funding data that has been generated so far. However, something came across my transom recently that is a place to start.

The NIH issued RFA-MD-21-004 Understanding and Addressing the Impact of Structural Racism and Discrimination on Minority Health and Health Disparities last year and the resulting projects should be on the RePORTER books by now. I was cued into this by a tweet from the Constellation Project which is something doing co-author networks. That may be useful for a related issue, that of collaboration and co-work. For now, I’m curious about what types of PIs have been able to secure funding from this mechanism. According to my RePORTER search for the RFA, there are currently 17 grants funded.

Of the funded grants, there are 4 from NIMHD, 4 from NIDA, 2 from NIA, 1 each from NIMH, NIHNDS, NINR, NICHD, NIGMS, NIDCD, and NCCIH. In the RFA, NIMHD promised 6-7 awards, NIDA 2, NIA 6, NIGMS 4-6 so obviously NIDA overshot their mark, but the rest are slacking. One each was promised for NIMH, NINDS, NICHD, NIDCD and NCCIH, so all of these are on track. Perhaps we will see a few more grants get funded by the time the FY elapses on Sept 30.

So who is getting funded under this RFA? Doing a quick google on the PIs, and admittedly making some huge assumptions based on the available pictures, I come up with

PI/Multi-PI Contact: White woman (2 NIA; 1 NCCIH; 3 NIDA; 1 NIDCD; 1 NIGMS; 1 NINDS); Black woman (1 NIDA; 1 NICHD; 1 NIMHD); Asian woman (1 NIMHD; 1 NIMHD; 1 NINR); White man (1 NIMHD; 1 NIMH)

Multi-PI, non-contact: Asian woman (1 NIA, 1 NIDA, 1 NIMHD); Black woman (2 NIDA, 1 NIMHD); White woman (1 NIDCD; 1 NIGMS; 1 NINR) Black man (1 NIGMS; 1 NIMH); White man (2 NIMH)

I would say the place I am most likely to be off in terms of someone who appears to me to be white but identifies as a person of color would be white women. Maybe 2-3 I am unsure of. I didn’t bother to keep track of how many of the non-contact PIs are on the proposals with white Contact PIs versus the other way around but….I can’t recall seeing even one where a non-contact white PI was on a proposal with a contact PI who is Black or Asian. (There was one award with three white men and one Black man as PIs and, well, does anyone get away with a four PI list that includes no woman anymore?) Anyway… make of that what you will.

I suspect that this RFA outcome is probably slightly better than the usual? And that if you looked at NIH’s studies that deal with communities or color and/or their health concerns more generally it would be even more skewed towards white PIs?

Ginther et al 2011 reported 69.9% of apps in their sample had white PIs, 16.2% had Asian PIs and 1.4% had Black PIs. Hoppe et al 2019 reported (Table S1) 1.5% of applications had Black PIs and 65.7% had white PIs in their original sample. So the 11 out of 17 grants having white PIs/Contact MultiPIs matches expected distribution, as does 3 Asian PIs. Black PIs are over represented since 1-2% of 17 is..zero grants funded. So this was not an opportunity that NIH took to redress the Ginther Gap.

But should it be? What should be the identity of PIs funded to work on issues related to “racism and discrimination” as it applies to “minority health and health disparities”? The “best” as determined by a study section of peer scientists, regardless of applicant characteristics? Regardless of the by now very well established bias against applications with Black PIs?

Someone on twitter asked about the panel that reviewed these grants. You can see from the funded grants on RePORTER that the study section reviewing these proposals was ZMD1 KNL (J1). Do a little web searching and you find that the roster for the 11/15/2021-11/17/2021 meeting is available. A three day meeting. That must have been painful. There are four chairs and a huge roster listed. I’m not going to search out all of them to figure out how many were white on the review panel. I will note that three of the four chairs were white and one was Asian (three of four were MDs, one was a PHD). This is a good place for a reminder that Hoppe et al reported 2.4% of reviewers were Black and 77.8% white in the study sections reviewing proposals for funding in FY2011-2015. I would be surprised if this study section was anything other than majority white.

Research Opportunities for New and “At-Risk” Investigators to Promote Workforce Diversity

NIDA, NIMH, and NINDS have issued a Program Announcement (PAR-22-181) to provide Research Opportunities for New and “At-Risk” Investigators with the intent to Promote Workforce Diversity.

This is issued as a PAR, which is presumably to allow Special Emphasis Panels to be convened. It is not a PAS, however, the announcement includes set-aside funding language familiar to PAS and RFA Funding Opportunity Announcements (FOA).

Funds Available and Anticipated Number of Awards The following NIH components intend to commit the following amounts for the duration of this PAR: NINDS intends to commit up to $10 million per fiscal year, approximately 25 awards, dependent on award amounts; NIDA intends to commit up to $5 million per fiscal year, 12-15 awards, dependent on award amounts; NIMH intends to commit up to $5 million per fiscal year, 12-15 awards, dependent on award amounts; Future year amounts will depend on annual appropriations.

This is a PA typical 3 year FOA which expires June 7, 2025. Reciept dates are one month ahead of standard, i.e., Sept (new R01) / Oct (Resub, Rev, Renew); Jan/Feb; May/Jun for the respective Cycles.

Eligibility is in the standard categories of concern including A) Underrepresented Racial/Ethnic groups, B) Disability, C) economic disadvantage and D) women. Topics of proposal have to be within the usual scope of the participating ICs. Eligibility of PIs is for the familiar New Investigators (“has not competed successfully for substantial, NIH (sic) independent funding from NIH“) and a relatively new “at risk” category.

At risk is defined as “has had prior support as a Principal Investigator on a substantial independent research award and, unless successful in securing a substantial research grant award in the current fiscal year, will have no substantial research grant funding in the following fiscal year.“

So. We have an offset deadline (at least for new proposals), set aside funds, SEPs for review and inclusion of NI (instead of merely ESI) and the potential for the more experienced investigator who is out of funding to get help as well. Pretty good! Thumbs up. Can’t wait to see other ICs jump on board this one.

To answer your first question, no, I have no idea how this differs from the NINDS/NIDA/NIAAA NOSI debacle. As a reminder:

Notice NOT-NS-21-049 Notice of Special Interest (NOSI): NIH Research Project Grant (R01) Applications from Individuals from Diverse Backgrounds, Including Under-Represented Minorities was released on May 3, 2021.

The “debacle” part is that right after NIDA and NIAAA joined NINDS in this NOSI, the Office of the Director put it about that no more ICs could join in and forced a rescinding of the NOSI on October 25, 2021 while claiming that their standard issue statement on diversity accomplished the same goals.

I see nothing in this new PAR that addresses either of the two real reasons that may have prompted the Office of the Director to rescind the original NOSI. The first and most likely reason is NIH’s fear of right wing, anti-affirmative action, pro-white supremacy forces in Congress attacking them. The second reason would be people in high places* in the NIH that are themselves right wing, anti-affirmative action and pro-white supremacy. If anything, the NOSI was much less triggering since it came with no specific plans of action or guarantees of funding. The PAR, with the notification of intended awards, is much more specific and would seemingly be even more offensive to right wingers.

I do have two concerns with this approach, as much as I like the idea.

First, URM-only opportunities have a tendency to put minority applicants in competition with each other. Conceptually, suppose there is an excellent URM qualified proposal that gets really high priority scores from study section and presume it would have also done so in an open, representation-blind study section. This one now displaces another URM proposal in the special call and *fails to displace* a lesser proposal from (statistically probable) a majoritarian PI. That’s less good than fixing the bias in the first place so that all open competitions are actually open and fair. I mentioned this before:

These special FOA have the tendency to put all the URM in competition with each other. This is true whether they would be competitive against the biased review of the regular FOA or, more subtly, whether they would be competitive for funding in a regular FOA review that had been made bias-free(r). […] The extreme example here is the highly competitive K99 application from a URM postdoc. If it goes in to the regular competition, it is so good that it wins an award and displaces, statistically, a less-meritorious one that happens to have a white PI. If it goes in to the MOSAIC competition, it also gets selected, but in this case by displacing a less-meritorious one that happens to have a URM PI. Guaranteed.

The second concern is one I’ve also described before.

In a news piece by Jocelyn Kaiser, the prior NIH Director Elias Zerhouni was quoted saying that study sections responded to his 2006/2007 ESI push by “punishing the young investigators with bad scores”. As I have tried to explain numerous times, phrasing this as a matter of malign intent on the part of study section members is a mistake. While it may be true that many reviewers opposed the idea that ESI applicants should get special breaks, adjusting scores to keep the ESI application at the same chances as before Zerhouni’s policies took effect is just a special case of a more general phenomenon.

So, while this PAR is a great tactical act, we must be very vigilant for the strategic, long term concerns. It seems to me very unlikely that there will be enthusiasm for enshrining this approach for decades (forever?) like the ESI breaks on merit scores/percentiles/paylines. And this approach means it will not be applied by default to all qualifying applications, as is the case for ESI.

Then we get to the Oppression Olympics, an unfortunate pitting of the crabs in the barrel against each other. The A-D categories of under-representation and diversity span quite a range of PIs. People in each category, or those who are concerned about specific categories, are going to have different views on who should be prioritized. As you are well aware, Dear Reader, my primary concern is with the Ginther gap. As you are aware, the “antis” and some pro-diversity types are very concerned to establish that a specific person who identifies as African-American has been discriminated against and is vewwwwy angweee to see any help being extended to anyone of apparent socio-economic privileges who just so happens to be Black. Such as the Obama daughters. None of us are clean on this. Take Category C. I have relatively recently realized that I qualify under Category C since I tick three of the elements, only two are required. I do not think that there is any possible way that my qualification on these three items affects my grant success in the least. To do so would require a lot of supposing and handwaving. I don’t personally think that anyone like me who qualifies technically under Category C really should be prioritized against, say, the demonstrated issue with the Ginther gap. These are but examples of the sort of “who is most disadvantaged and therefore most deserving” disagreement that I think may be a problem for this approach.

Why? Because reviewers will know that this is the FOA they are reviewing under. Opinions on the relative representation of categories A-D, Oppression Olympics and the pernicious stanning of “two-fers” will be front and present. Probably explicit in some reviews. And I think this is a problem in the broader goals of improving equity of opportunity and in playing for robust retention of individuals in the NIH funded research game.

__

*This is going to have really ugly implications for the prior head of the NIH, Francis Collins, if the PAR is not rescinded from the top and the only obvious difference here is his departure from NIH.

Grant awards and the new, new NIH Biosketch

Way back in 2015 the NIH made some major changes to the Biosketch. As detailed in this post, one of the major changes was replacing the long list of publications with a “contribution to science” section which was supposed to detail up to five areas of focus with up to four papers, or other research products, cited for each contribution. Some of the preamble from NIH on this suggests it was supposed to be an anti-Glamour measure. Sure. There was also an inclusion of a “personal statement” which was supposed to be used to further brag on your expertise as well as to explain anything…funny… about your record.

In dealing with the “contributions to science” change, I more or less refused to do what was requested. As someone who had been a PI for some time, had mostly senior author pubs and relatively few collaborative papers, I could do this. I just made a few statements about an area I have worked in and listed four papers for each. I didn’t describe my specific role as instructed. I didn’t really describe the influence or application to health or technology. So far this has gone fine, as I can’t remember any comments on Investigator on grants I’ve submitted with this new (old) Biosketch that appear confused about what I have done.

The NIH made some other changes to the Biosketch in 2021, the most notable of which was the removal of the list of Research Support that was previously in Section D. I pointed out in a prior post that I suspect this was supposed to be an attempt to break a specific culture of peer review. One that had hardened reviewers and applicants against the longstanding intent of the NIH. It is very clear in the prior instructions that Section D was not supposed to list all active and completed funding over the past three years. The NIH instructed us to only include that which we wanted to call attention to and added the note that it was for reviewers to assess qualifications of the research team for the new project being proposed. They further underlined this by instructing applicants not to confuse this with the Other Support page which was explicitly for reporting all funding. This failed entirely.

As we have said many times, many ways on this blog…. woe betide any poor newbie applicant who takes the instructions about other grant support at face value and omits any funding that can be easily found on funder websites or the investigator’s lab or University splash page. Reviewers will get in a high dudgeon if they think the PI is trying to conceal anything about their research support. This is, I will assert, because they either overtly or covertly are interested in two things. Neither of which the NIH wants them to be interested in.

One, does the PI have “too much money” in their estimation. The NIH is absolutely opposed to reviewers letting their evaluation of proposal merit be contaminated with such concerns but….people are people and jealously reigns supreme. As does self-righteous feelings about how NIH funds should be distributed. So…review, in practice, is biased in a way that the NIH does not like.

The second concern is related, but is about productivity and is therefore slightly more palatable to some. If the recitation of funding is selective, the PI might be motivated to only present projects that have been the most productive or led to the most Glammy papers. They might be also motivated to omit listing any project which have, by some views, under-produced. This is a tricky one. The instructions say reviewers will look at what the PI chooses to list on the Biosketch as evidence of their overall qualifications. But. How can a reviewer assess qualifications only from the projects that went amazingly well without also assessing how many tanked, relatively speaking? Or so would think a reviewer. The NIH is a little more wobbly on this one. “Productivity” is a sort-of tolerated thing and some analysis of papers-per-grant-dollar (e.g. from NIGMS) show their interest, at least from a Program policy perspective. But I think overall that Program does not want this sort of reviewer bean counting to contaminate merit review too much- the Biosketch instructions insist that the direct costs should not be included for any grants that are mentioned. Program wants to make the calls about “too much money”.

Ok so why am I blogging this again today? Well, we’re into the second year of the new, new attempt of NIH to get the list of grants on the Biosketch more selective. And I’m thinking about how this has been evolving in grants that I’ve been asked to review. Wait..”more selective”? Oh yes, the list of grants can now be added to Section A, the Personal Statement. With all of the same language about how this is only for ongoing or completed projects “that you want to draw attention to“. NOT-OD-21-073 even ties this new format description to the re-organization of the Other Support page, again making it clear that these are not the same thing.

So the question of the day is, how are applicants responding? How are reviewers reacting to various options taken by applicants?

I put in my first few applications with the grant list simply removed. I added a statement to Section A summarizing my total number of intervals of competitive support as PI and left it at that. But I’ve seen many applicants who put all their grants in Section A, just as they would have put them in Section D before.

I guess I had better do the same?

Program Note

I’ve just tried to import most of the content from the Scientopia version of Drugmonkey into this, my original blog host/site. I’d done some importing after ScienceBlogs cashiered me and maybe once after we started Scientopia. But it looked like nothing had been added since 2014. So I addressed that.

Blogs are dead. Yes. As some of you may have noticed we were having trouble with the Scientopia install over the past several years, what with it periodically leading to browser blocks and security warnings. This was in the wake of a change of hosts that included messing up the appearance with ads and and some formatting changes that I was unable to rapidly fix. It has been a labor of love for some Scientopians behind the scenes but in essence, nobody is really taking care of it. We have no $ for hosting, the nascent attempts at ads never really generated enough to cover the bills. As you may know I have steadfastedly resisted being anything other than content, particularly when there were some who wanted to be involved but weren’t really blogging much and my content was driving most of the traffic.

I have decided it is time to import my blather back to wordpress.com to facilitate my linking to old posts on the twitter and to stave off the inevitable, when Scientopia goes dark and the domain is bought by some spam farm.

I expect that if I do blog now and again it will be here on the old site.

ashes to ashes and dust to dust.

Reconsidering “Run to Daylight” in the Context of Hoppe et al.

In a prior post, A pants leg can only accommodate so many Jack Russells, I had elucidated my affection for applying Vince Lombardi’s advice to science careers.

Run to Daylight.

Seek out ways to decrease the competition, not to increase it, if you want to have an easier career path in academic science. Take your considerable skills to a place where they are not just expected value, but represent near miraculous advance. This can be in topic, in geography, in institution type or in any other dimension. Work in an area where there are fewer of you.

This came up today in a discussion of “scooping” and whether it is more or less your own fault if you are continually scooped, scientifically speaking.

If yr getting "scooped" alla the time, it is perhaps worth interrogating how you choose what scientific lines of inquiry to pursue.

— @mnitabach.bsky.social (@mnitabach) January 7, 2022

He’s not wrong. I, obviously, was talking a similar line in that prior post. It is advisable, in a career environment where things like independence, creativity, discovery, novelty and the like are valued, for you NOT to work on topics that lots and lots of other people are working on. In the extreme, if you are the only one working on some topic that others who sit in evaluation of you see as valuable, this is awesome! You are doing highly novel and creative stuff.

The trouble is, that despite the conceits in study section review, the NIH system does NOT tend to reward investigators who are highly novel solo artists. It is seemingly obligatory for Nobel Laureates to complain about how some study section panel or other passed on their grant which described the plans to pursue what became the Nobel-worthy work. Year after year a lot of me-too grants get funded while genuinely new stuff flounders. The NIH has a whole system (RFAs, PAs, now NOSI) set up to beg investigators to submit proposals on topics that are seemingly important but nobody can get fundable scores to work on.

In 2019 the Hoppe et al. study put a finer and more quantitatively backed point on this. One of the main messages was the degree to which grant proposals on some topics had a higher success rate and some on other topics had lower success rates. You can focus on the trees if you want, but the forest is all-critical. This has pointed a spotlight on what I have taken to calling the inherent structural conservatism of NIH grant review. The peers are making entirely subjective decisions, particularly right at the might-fund/might-not-fund threshold of scoring, based on gut feelings. Those peers are selected from the ranks of the already-successful when it comes to getting grants. Their subjective judgments, therefore, tend to reinforce the prior subjective judgments. And of course, tend to reinforce an orthodoxy at any present time.

NIH grant review has many pseudo-objective components to it which do play into the peer review outcome. There is a sense of fair-play, sauce for the goose logic which can come into effect. Seemingly objective evaluative comments are often used selectively to shore up subjective, Gestalt reviewer opinions, but this is in part because doing so has credibility when an assigned reviewer is trying to convince the other panelists of their judgment. One of these areas of seemingly objective evaluation is the PI’s scientific productivity, impact and influence, which often touches on publication metrics. Directly or indirectly. Descriptions of productivity of the investigator. Evidence of the “impact” of the journals they publish in. The resulting impact on the field. Citations of key papers….yeah it happens.

Consideration of the Hoppe results, the Lauer et al. (2021) description of the NIH “funding ecology” in the light of some of the original Ginther et al. (2011, 2018) investigation into the relationship of PI publication metrics is relevant here.

Publication metrics are a function of funding. The number of publications a lab generates depend on having grant support. More papers is generally considered better, fewer papers worse. More funding means an investigator has the freedom to make papers meatier. Bigger in scope or deeper in converging evidence. More papers means, at the very least, a trickle of self-cites to those papers. More funding means more collaborations with other labs…which leads to them citing both of you at once. More funding means more trainees who write papers, write reviews (great for h-index and total cites) and eventually go off to start their own publication records…and cite their trainee papers with the PI.

So when the NIH-generated publications say that publication metrics “explain” a gap in application success rates, they are wrong. They use this language, generally, in a way that says Black PIs (the topic of most of the reports, but this generalizes) have inferior publication metrics so this causes a lower success rate. With the further implication that this is a justified outcome. This totally ignores the inherent circularity of grant funding and publication measures of awesomeness. Donna Gither has written a recent reflection on her work on NIH grant funding disparity, which doubles down on her lack of understanding on this issue.

Publication metrics are also a function of funding to the related sub-field. If a lot of people are working on the same topic, they tend to generate a lot of publications with a lot of available citations. Citations which buoy up the metrics of investigators who happen to work in those fields. Did you know, my biomedical friends, that a JIF of 1.0 is awesome in some fields of science? This is where the Hoppe and Lauer papers are critical. They show that not all fields get the same amount of NIH funding, and do not get that funding as easily. This affects the available pool of citations. It affects the JIF of journals in those fields. It affects the competition for limited space in the “best” journals. It affects the perceived authority of some individuals in the field to prosecute their personal opinions about the “most impactful” science.

That funding to a sub-field, or to certain approaches (technical, theoretical, model, etc, etc) has a very broad and lasting impact on what is funded, what is viewed as the best science, etc.

So is it good advice to “Run to daylight”? If you are getting “scooped” on the regular is it your fault for wanting to work in a crowded subfield?

It really isn’t. I wish it were so but it is bad advice.

Better advice is to work in areas that are well populated and well-funded, using methods and approaches and theoretical structures that everyone else prefers and bray as hard as you can that your tiny incremental twist is “novel”.

The SABV Camel and the NIH tent flap

You know the old story.

In this new story, we have the NIH’s Sex As a Biological Variable (SABV) policy. When first discussed, just about everyone who took this seriously pointed out the problem of a zero sum, limited funding system adopting a mandate which would double the animal costs. To really consider SABV properly, we said, this is going to double our sample sizes…at the very least. Probably more than double.

That is coming from the perspective of a scientist who works with units of the whole experimental animal. There are many of us.

The official NIH response was a bunch of gaslighting.

“Oh no”, went the policy mavens of the NIH, “this is not what this means at all. Simply include equal numbers of male and female animals at your regular sample size. That’s it. Oh, yeah, you have to say you will stratify your data by sex and look at it. You know, just in case there’s anything there. But nothing insists you have to double your sample size.”

Sure, said we NIH watchers/applicants. Sure it will go like that. Have you met our reviewers? They are going to first of all demand that every study is fully powered to detect any sex difference. Then, they are going to immediately start banging on about swabbing and cycling the female rats and something something about powering up for cycle as well.

NIH: “No, of course not that would never happen why we will tell them not to do that and everything will be copacetic”

Things were not copacetic. As predicted, reviewers of grants have, since even before the mandate went into effect, demonstrated they are constitutionally unable to do what NIH claimed they should be doing and in fact do what they were predicted in advance to do. Make everything HAVE to be a sex differences study and HAVE to be a study of estrous cycle. Randomly. Variable. Yes. As with everything in NIH review. And who knows, maybe this is a selective cudgel (I call it Becca’s Bludgeon) used only when they just generally dislike the proposal.

The NIH mandate let the SABV camel’s nose under the tentflap and now that camel is puuuuuuuuussssshhhhing all the way in.

A new article in eLife by Garcia-Sifuentes and Maney is part of this campaign. It is chock full of insinuations and claims trying to justify the full camel in side the tent. Oh, they know perfectly well what the NIH policy was. But they are using all of the best #allegedprofession techniques to try to avoid admitting they are fully doing an end run.

From the Abstract: This new policy has been interpreted by some as a call to compare males and females with each other.

From the Intro: Although the NIH policy does not explicitly require that males and females be compared directly
with each other, the fact that more NIH-funded researchers must now study both sexes should lead to an increase in the frequency of such comparisons (insert self-citation). For example, there should be more testing for sex-specific
responses

“should”.

although the proportion of articles that included both sexes significantly increased (see also Will et al., 2017), the proportion that treated sex as a variable did not. [Note interesting goalpost move. or at least totally undefined insinuation] This finding contrasts sharply with expectations [whose “expectations” would those be?], given not only the NIH mandate but also numerous calls over the past decade to disaggregate all preclinical data by sex [yes, the mandate was to disaggregate by sex. correct.] and to test for sex differences [bzzzt, nope. here’s another slippery and dishonest little conflation]

One potential barrier to SABV implementation is a lack of relevant resources; for example, not all researchers have received training in experimental design and data analysis that would allow them to test for sex differences using appropriate statistical approaches. [oh what horseshit. sure, maybe there is a terrible lack of experimental design training. I agree those not trained in experimental psychology seem to be a bit lacking. But this is not specific to sex differences. A group is a group is a group. so is a factor. the “lack of relevant resources” is….money. grant money.]

any less-than-rigorous test for sex differences creates risk for misinterpretation of results and dissemination of misinformation to other scientists and to the public [There you have it. The entire NIH scheme to introduce SABV is not only flawed, it is, seemingly, even worse than doing nothing!]

Although a sex difference was claimed in a majority of articles (57%), not all of these differences were supported with statistical evidence. In more than a quarter of the articles reporting a sex difference, or 24/83 articles, the sexes were never actually compared statistically. [Yep, totally consistent with the assertions from NIH about what they were after. Anything else is a significant move of the goalposts. In the direction that was anticipated and EXPLICITLY denied as being the goal/end game by the NIH. In oh so many ways.]

In these cases, the authors claimed that the sexes responded differentially to a treatment when the effect of treatment was not statistically compared across sex. … Of the studies with a factorial design, 58% reported that the sexes responded differently to one or more other factors. The language used to state these conclusions often included the phrase ‘sex difference’ but could also include ‘sex-specific effect’ or that a treatment had an effect ‘in males but not females’ or vice versa. … Neither approach tests whether the treatment had different effects in females and males. Thus, a substantial majority of articles containing claims of sex-specific effects (70%) did not present statistical evidence to support those claims

This is also, utter a-scientific horseshit.

I get this a lot from reviewers so I’m going to expand but only briefly. There is no such thing as canonical statistical interpretation techniques that are either “right” or “wrong”. Nor do statistical inference techniques alter the outcome of a study. The data are what they are. All else is shades of interpretation. At the very best you could say that different inferential statistical outcomes may mean there is stronger or weaker evidence for your interpretations of the data. at best.

But there is a broader hypocrisy here. Do you only build your knowledge within the context of one paper? Do you assemble your head space on whether something is likely or unlikely to be a valid assertion (say, “female rats self-administer more cocaine”) ONLY on papers that provide like-to-like perfectly parallel and statistically compared groups?

If you are an idiot, I suppose. Not being an idiot, I assert that most scientists build their opinions about the world of science that they inhabit on a pile of indirectly converging evidence. Taking variability in approach into account. Stratifying the strength of the evidence to their best ability. Weighting the results. Adding each new bit of evidence as they come across it.

And, in a scenario where 10 labs were conducting cocaine self-administration studies and five each tended to work on males and females independently, we would conclude some things. If we were not preening Experimental Design Spherical Cow 101 idiots. If, for example, no matter the differences in approach it appeared that in aggregate the females self-administered twice as many infusions of the cocaine.

We would consider this useful, valid information that gives us the tentative idea that perhaps there is a sex difference. We would not hold our hands over our eyes mumbling “blah blah blah I can’t hear you either” and insist that there is zero useful indication from this true fact. We would, however, as we do with literally every dataset, keep in mind the limitations of our inferences. We might even use these prior results to justify a better test of the newly developed hypothesis, to overcome some of the limitations.

That is how we build knowledge.

Not by insisting if a comparison of datasets/findings does not accord with strict ideas of experimental design rigor, it is totally invalid and meaningless.

Among the articles in which the sexes were pooled, the authors did so without testing for a sex difference almost half of the time (48%; Figure 3B). When authors did test for a sex difference before pooling, they sometimes found a significant difference yet pooled the sexes anyway; this occurred in 17% of the articles that pooled.[Yes, consistent with the NIH policy. Again with the moving the goalposts….]

Thus, the authors that complied with NIH guidelines to disaggregate data usually went beyond NIH guidelines to explicitly compare the sexes with each other. [hookay…..so where’s the problem? isn’t this a good thing?]

Peer review is merely advisory to the decision maker

How many times have you heard another academic scientist say “I rejected that manuscript…“. Or, “I accepted that manuscript….“? This is usually followed by some sort of criticism of an outcome for that manuscript that is inconsistent with their views on what the disposition should be. Most often ” I rejected that manuscript…but it was accepted for publication anyway, how dare they??!!??”

We somewhat less often hear someone say they “rejected” or “funded” a grant proposal…but we do hear disappointed applicants claim that one reviewer “killed my grant”.

This is, in general, inaccurate.

All, and I mean ALL, of the review input on NIH grants that takes place from receipt and referral through to the Advisory Council input (and whatever bean counting tetris puzzle fitting happens post-Council) is merely advisory to the Director. The IC Director is the deciderer.

Similarly, all peer review input to manuscripts is merely advisory to the Editor. In this case, there may be some variability in whether it is all being done at the Editor in Chief level, to what extent she farms that out to the handling sub-Editors (Associate, Senior, Reviewing, etc) or whether there is a more democratic discussion amongst a group of deciding editors.

What is clear, however, is that the review conducted by peers is merely advisory.

It can be the case that the deciding editor (or editorial process) sides with a 2-1 apparent vote. It could be siding with a 1-2 vote. Or overruling a 0-3 vote. Either for or against acceptance.

This is the process that we’ve lived with for decades. Scientific generations.

Yet we still have people expressing this bizarre delusion that they are the ones “accepting” or “rejecting” manuscripts in peer review. Is this a problem? Would it be better, you ask, if we all said “I recommended against accepting it”?

Yes. It would be better. So do that.

This post is brought to you by a recent expression of outrage that a paper was rejected despite (an allegation of) positive-sounding comments from the peer reviewers. This author was so outraged that they contacted some poor fool reviewer who had signed their name to the review. Outside of the process of review, the author demanded this reviewer respond. Said reviewer apparently sent a screen shot of their recommendation for, well, not rejection.

This situation then usually goes into some sort of outrage about how the editorial decision making process is clearly broken, unethical, dishonest, political….you know the drill. Bad.

For some reason we never hear those sorts of complaints from the authors when an editor has overruled the disfavorable reviewers and issued an acceptance for publication.

No, in those cases we hear from the outraged peer reviewer. Who also, on occasion, has been know to rant about how the editorial decision making process is clearly broken, unethical, dishonest, political….you know the drill. Bad.

All because we have misconstrued the role of peer review.

It is advisory. That is all.

Really, Dartmouth College Psych Dept? Really???

The Department of Psychological and Brain Sciences at Dartmouth College (yes, that department) has announced a most unusual academic position that they are seeking to fill.

The @Dartmouth Department of Psychological and Brain Sciences invites applications for a Faculty Fellow: a postdoc appointment that will *automatically* convert to a regular full-time tenure track assistant professorship after 2 years! Details: https://t.co/uYOFSND25s

— Dartmouth Psychological and Brain Sciences (@DartmouthPBS) September 21, 2021

Huh? Well, let us click through and read the details.

The Department of Psychological and Brain Sciences at Dartmouth College invites applications for a Faculty Fellow, a two-year residential postdoctoral appointment, that will convert automatically to a regular full-time tenure-track appointment as Assistant Professor. Faculty Fellows are part of a cohort of faculty committed to increasing diversity in their disciplines. We are interested in applicants whose research can connect to and/or bridge between any foci in our department including behavioral, cognitive, social and affective psychology and neuroscience. We are especially interested in candidates who have a demonstrated ability to contribute to Dartmouth’s diversity initiatives in STEM research, such as the Women in Science Program, E. E. Just STEM Scholars Program, and Academic Summer Undergraduate Research Experience (ASURE).

That’s about it. The rest is boilerplate with minimal details, including about the salary and resources being offered. So we have to assume “postdoctoral appointment” means the usual for a postdoc. Salary something along the lines of the NRSA scale and no individual resources such as a nice startup package or research space. Maybe there will be, but it is not on evidence in the job solicitation.

This is CLEARLY a DEI hire. An attempt to diversity a faculty that looks to my eye like it could use some diversification.

Instead of just hiring a person at the faculty level directly, they will be getting faculty level effort and behavior out of someone for the low, low price of a postdoc stipend. With a guarantee of “automatic” conversion which one, frankly, doubts will be iron clad.

This is so dismally emblematic of the institutional efforts to respond to the pressure to diversify their faculty.

Are any of you seeing similar proposals lauched at your University? What is the rationale here? What is the justification for this over just creating new faculty lines and hiring into them?

I can think of a couple of rationalescuses.

It is some sort of tenure clock manipulation. If they think, for whatever reason, that someone that will be able to contribute to “Dartmouth’s diversity initiatives in STEM research” will have a hard time making tenure on the usual schedule in their Department, this could be the reason for the plan. This would be an extra red flag warning to any applicant, of course. If the Department can’t get behind valuing these contributions as a substitute for their other expectations, and only see them as add-on effort that delays “real progress”, then this person will always be at odds with an unsupportive Department. Tenure is a risky proposition, no matter how long the decision is delayed.

It could be some sort of “we can get this approved quickly but oh how hard it is to get a new tenure line approved for this cycle” thing. Yeah, well that questions the commitment of the College and the “automatic” conversion. So surely this isn’t going to be raised.

A colleague from elsewhere indicated that something like this is being tried in their Department. The rationale is, from what I can tell, that scientist of color are reluctant to take on postdoctoral training (pretty sure I’ve seen data on that mentioned somewhere) and that this leak in the pipeline could be addressed by offering faculty positions earlier. Ok, I definitely buy that more security of a career would be helpful to keep promising younger scientists from bailing on the academic track before or during the expected postdoctoral interval. But. But, but but. Why not just hire straight into a faculty position? Course relief, service relief, etc, is already standard operating procedure. If a Department or University (or College) thinks this needs to be extended two or three years longer for these earlier-career hires, so be it.

This brings us to the longer arc of wage manipulation in the individual sense and in the industry sense.

If these Departments who are all really concerned about DEI and are launching various hiring initiatives were serious, they would have to be out there competing with each other for the existing pool of academic scientists in more or less the same position as their usual hires. As we know, there aren’t a lot of them, particularly when it comes to African-American scientists and some other key Federally defined underrepresented groups. So, according to market forces the Departments would have to PAY. More salary. More support. More startup cash. More housing / relocation allowances. More spousal hire opportunity. More everything.

This plan short circuits that by locking in candidates before they are as competitive on the open market. When they are still relatively desperate and/or think this is a great opportunity to jump ahead on the career arc. And as more Departments catch wind of this excellent strategy they are more likely to opt for this can-kicking strategy and less likely to PAY to get those who are currently trained to the usual point of faculty hires.

Biased objective metrics

As you know, Dear Reader, one of the things that annoys me the most is being put in the position of having to actually defend Glam, no matter how tangentially. So I’m irritated.

Today’s annoyance is related to the perennial discussion of using metrics such as the Journal Impact Factor of journals in which a professorial candidate’s papers are published as a way to prioritize them for a job search. You can add h-index and citations of the candidate’s papers on an individual basis on this heap if you like.

The Savvy Scientist in these discussions is very sure that since these measures, ostensibly objective, are in fact subject to “bias”, this renders them risible as useful decision criteria.

We then typically downshift to someone yelling about how the only one true way to evaluate a scientist is to READ HER PAPERS and make your decisions accordingly. About “merit”. About who is better and who is worse as a scientist. About who should make the short list. About who should be offered employment.

The Savvy Scientist may even demonstrate that they are a Savvy Woke Scientist by yelling about how the clear biases in objective metrics of scientific ability and accomplishment work to the disfavor of non-majoritarians. To hinder the advancement of diversity goals by under-counting the qualities of URM, women, those of less famous training pedigree, etc.

So obviously all decisions should be made by a handful of people on a hiring committee reading papers deeply and meaningfully offering their informed view on merit. Because the only possible reason that academic science uses those silly, risibly useless, so called objective measures is because everyone is too lazy to do the hard work.

What gets lost in all of this is any thinking about WHY we have reason to use objective measures in the first place.

Nobody, in their Savvy Scientist ranting, seems to every consider this. They fail to consider the incredibly biased subjectivity of a handful of profs reading papers and deciding if they are good, impactful, important, creative, etc, etc.

Even before we get to the vagaries of scientific interests, there are hugely unjustified interpersonal biases in evaluating work products. We know this from the studies where legal briefs were de/misidentified. We can infer this from various resume-call back studies. We can infer this from citation homophily studies. Have you not every heard fellow scientists say stuff like “well, I just don’t trust the work from that lab”? or “nobody can replicate their work”? I sure have. From people that should know better. And whenever I challenge them as to why….let us just say the reasons are not objective. And don’t even get me started about the “replication crisis” and how it applies to such statements.

Then, even absent any sort of interpersonal bias, we get to the vast array of scientific biases that are dressed up as objective merit evaluations but really just boil down to “I say this is good because it is what I am interested in”. or “because they do things like I do”>

Citations metrics are an attempt to crowd source that quality evaluation so as to minimize the input of any particular bias.

That, for the slower members of the group, is a VERY GOOD THING!

The proper response to an objective measure that is subject to (known) biases is not to throw the baby out onto the midden heap of completely subjective “merit” evaluation.

The proper response is to account for the (known) biases.

Making sense of the T32 effect on the NIH funding disparity

A new blog post from Mike Lauer, Deputy Director for Extramural Research at the NIH, presents new data on grant award (K and R01equivalent) to applicants assuming they had a prior appointment on a NIH funded T32 training grant. This particular post included some demographic analysis which addressed URM vresus majoritarian status. Leaving aside for a moment the obvious concerns about who gets selected for T32 appointments and where and with whom they are correspondingly training, I was curious about the relative advantage.

Now, the first thing you will recognize is that the critical data are collapsed across all URM groups, although this post doesn’t seem to specify if Asian is included in non-URM, the total N in Table 1 and Table 3 suggest this is the case. This is a little disappointing of course, since the original Ginther report found such striking differences in award rates across URM groups. There was an Asian PI disparity and a Hispanic PI lackthereof, so would it not be super important to keep these groups separate for this post-T32 analysis? Of course it would.

But what got me scratching my head was the presentation of the percentages in Table 3. The show, for example, that 12.9% of non-URM T32 trainees eventually became PI on a funded R01 whereas only 9.8% of URM trainees became PI on a funded R01. That’s all well and good but it obscures the success rate because it doesn’t account for the differential submission rate (24.5% of nonURM T32 trainees eventually submitted an R01, 21.8% of URM trainees).

Doing a little math here, I make it out to be a 52.9% award rate for non-URM applicants and a 45.1% rate for URM applicants. This is a 7.8 percentage point differential which means the URM applicants have a success rate that is 85.3% of the non-URM applicant success rate.

Now we can put this in familiar terms. The Hoppe report found that Black applicants had a success rate that was 60.5% of that enjoyed by white applicants (it was 60.4 in Ginther et al, 2011). So if we take this 24.8 percentage point differential and divide it by the 39.5 percentage point deficit for Black applicants in the Hoppe sample…we end up with about 63% of the Black/white gap reported in Hoppe. The combination of T32 training history, combining of all URM together and adding Asian PIs to the non-URM group closes 63% of the gap.

See my problem here? We want all this broken down so we can answer the most simple question, does T32 training close the gap between Black and white PIs and if so to what extent.

THEN we can go on to ask about other URM populations and the effect of adding the Asian* T32 participants to the non-URM pile.

*reminder that Ginther found that applications with Asian PIs with domestic doctorates had success rates similar to those with white PIs. There is a major complication here with T32 eligibility for Asian-Americans versus Asians who were not T32 eligible as postdocs.

NOSI for URM Investigators: Choose Wisely

I was recently describing Notice NOT-NS-21-049 Notice of Special Interest (NOSI): NIH Research Project Grant (R01) Applications from Individuals from Diverse Backgrounds, Including Under-Represented Minorities in the context of the prior NIH Director’s comments that ESI scores got worse after the news got around about relaxed paylines.

One thing that I had not originally appreciated was the fact that you are only allowed to put one NOSI in Box 4b.

Which means that you have to choose. If you qualify as an individual from diverse backgrounds you could use this, sure. But that means you cannot use a NOSI that is specific to the topic you are proposing.

This is the usual NIH blunder of stepping on their own junk. How many ways can I count?

Look, the benefit of NOSI (and the predecessor, the Program Announcement) is uncertain. It seemingly only comes into play when some element of Program wishes to fund an award out of the order of review. Wait, you say, can’t they just do that anyway for whatever priority appears in the NOSI? Yes, yes they can….when it comes to the topic of the grant. So why do NOSI exist at all?

Well…one presumes it is because elements of Program do not always agree on what should be funded out of order of review. And one presumes there is some sort of conflict resolution process. During which the argument that one grant is related to the Programmatic Interest formally expressed in the NOSI has some weight or currency. Prioritizing that grant’s selection for funding over the identically-percentiled grant that does not mention a NOSI.

One still might wonder about a topic that fits the NOSI but doesn’t mention the NOSI directly. Well, the threat language at the bottom of some of those NOSI, such as oh I don’t know this one, is pretty clear to me.

• For funding consideration, applicants must include “NOT-DA-21-006” (without quotation marks) in the Agency Routing Identifier field (box 4B) of the SF424 R&R form. Applications without this information in box 4B will not be considered for this initiative.

Applications nonresponsive to terms of this NOSI will not be considered for the NOSI initiative.

So what is a PI to do? Presumably the NOSI has some non-negligible value and everyone is motivated to use those if possible. Maybe it will be the difference between a grey zone pickup and not, right? If your ideas for this particular grant proposal fit with something that your favorite IC has gone to the trouble if mentioning in a NOSI…well….dang it….you want to get noticed for that!

So what can you do if you are a person underrepresented who qualifies for the NOSI NOT-NS-21-049 ? The value of this one is uncertain. The value of any other NOSI for your particular application is likewise uncertain. We know perfectly well the NIH as a whole is running scared of right wing political forces when it comes to picking up grants. We know that this NOSI may be related to the well meaning ICs’ staff having difficulty getting PI demographic information and could simply be a data collection strategy for them.

Cynical you say? Well I had a few exchanges with a fairly high up Program person who suggested to me that perhaps the strategy was to sneak the “extra” NOSI into the Abstract of the proposal. This would somehow get it in front of Program eyes. But….but….there’s the “will not be considered” boilerplate. Right? What does this mean? It is absolutely maddening for PIs who might like to take advantage of this new NOSI which one might think would be used to fix the Ginther Gap. It is generally enraging for anyone who wants to see the Ginther Gap addressed.

It makes me positively incandescent to contemplate the possibility that the mere announcing of this NOSI will lead to study sections giving even worse scores to those applications, without any real assistance coming from Program.

A couple of more thoughts. This doesn’t apply to anything other than an R01 application, which is nuts. Why not apply it to all investigator initiated mechanisms? Trust me, underrepresented folks would like a leg up on R21 and R03 apps as well. These very likely help with later R01 getting, on a NIH wide statistical basis. You know, the basis of the Ginther Gap. So why not include other mechs?

And did you notice that no other ICs have joined? NINDS issued the NOT on May 3 and they were rapidly joined by NIDA (May 6) and NIAAA (May 11). All in good time for the June 5 and July 5 submission rounds. Since then….crickets. No other ICs have joined in. Weird, right?

I was on a Zoom thing awhile back where a highly authoritative Program person claimed that the Office of the Director (read: Francis Collins) had put a hold on any more ICs joining the NINDS NOSI.

Why? Allegedly because there was a plan to make this more general, more NIH wide all at one fell swoop.

Why? Who the heck knows. To cover up the reluctance of some of the ICs that would not be joining the NOSI if left up to their own devices? If so, this is HORRENDOUS, especially give the above mentioned considerations for choosing only one NOSI for Box 4b. Right? If they do extend this across all NIH, how would any PI know that their particular IC has no intention whatsoever of using this NOSI to fund some grants? So maybe they choose to use it, for no help, while bypassing another NOSI that might have been of use to them.

Good Mentoring

One of the recurring discussions / rants in academic circles is the crediting of good mentoring to the Professor. I’m not going to tag the stimulus of the day because it generalizes and because I have no idea what motivates any particular person on any particular day.

There does seem to be a common theme. A Professor, usually female and usually more-junior, is upset that her considerable efforts to mentor students or postdocs does not seem to get as much credit as they should. This is typically contextualized by oblique or specific reference that some other professors do not put in the effort to “mentor” their trainees as well and this is not penalized. Furthermore, there is usually some recognition that a Professor’s time is limited and that the shoddiness of the mentoring of those other peers lets them work on what “really counts”, i.e., papers and grants, to an advantage over the self-identified good mentor.

Still with me?

There is a further contribution of an accusation, implicit or explicit, that those other peer Professors are not just advantaged by not spending time on “mentoring” but also advantaged by doing anti-mentoring bad things to their trainees to drive them to high scientific/academic output which further advantages the bad mentor colleagues against our intrepid hero Good Mentor.

Over on Twitter I’ve been pursuing one of my usual conundrums as I try to understand the nature of any possible fixes we might put in place with regard to “good” and “bad” academic mentoring, i.e., the role of career outcome in influencing how the mentee and evaluating bodies might view the quality of mentoring practices. My point is that I’ve seen a lot of situations where the same PI is viewed as providing both a terrible and a good-to-excellent mentoring environment by different trainees. And the views often align with whether the trainee is satisfied or dissatisfied with their career outcomes, and align less well with any particular behaviors of the PI.

Here, I want to take up the nature of tradeoffs any given Professor might have, in the context of trying to mentor more than one academic trainee, yes concurrently, but also in series.

My assertion is that “good mentoring” takes time, it takes the expenditure of grant and other funds and it takes the expenditure of social capital, in the sense of opportunities. In the case of most of the Professoriate, these are all limited resources.

Let us suppose we have two graduate students nearing completion, in candidacy and up against a program requirement for, e.g., three published papers. Getting to the three published papers, assuming all else equal between the two trainees, can be greatly affected by PI throw down. Provision of assistance with drafting the manuscript, versus terse, delayed “markup” activities? Insisting the paper needs to get into a certain high JIF journal, versus a strategy of hitting solid society journals. Putting research dollars into the resources, capital or personnel, that are going to speed progress to the end, versus expecting the trainee to just put in more hours themselves.

A PI should be “fair”, right? Treat everyone exactly the same, right? Well…it is never that simple. Research programs have a tendency not to go to plan. Projects can appear to level themselves up or down after each experiment. Peer review demands vary *tremendously* and not only by journal JIF.

Let us suppose we have two postdocs nearing a push for an Assistant Professor job. This is where the opportunities can come into play. Suggesting a fill-in for a conference presentation. Proposing conference symposia and deciding which trainee’s story to include. Choosing which project to talk about when the PI is herself invited. Pushing collaborations. Manuscript review participation with a note to the Associate Editor. Sure, it could be “fair”. But this is a game of competitive excellence and tiny incremental improvements to the odds of landing a faculty position. Is it “good mentoring” if taking a Harrison Bergeron approach means you never seem to land any postdocs from the laboratory in the plummiest of positions? When a focal throwdown on one would mean they have a good chance but divide-and-conquer fails to advance anyone?

More importantly, the PIs themselves have demands on their own careers. “Aha”, you cry, “this selfishness is what I’m ON about.”. Well yes…..but how good is the mentoring if the PI doesn’t get tenure while the grad student is in year 3? Personal experience on that one, folks. “not good” is the answer. Perhaps more subtly, how is the mentoring going to be for the next grad student who enters the laboratory when the PI has generated “fair” publishing prior trainees but not the glamourous publications needed to land that next grant? How much better is it for a postdoc entering the job market when the PI has already placed several Assistant Professors before them?

Or, less catastrophically, what if the PI has expended all of the grant money on the prior student’s projects which the student constructed and just happens to be highly expensive (“my mentor supports my research (1-5”))? Is that good mentoring? Well yeah, for the lucky trainee but it isn’t fair in a serial sense, is it?

Another common theme in the “good mentor” debate is extending “freedom” to the trainee. Freedom to work on “their ideas”. This is a tough one. A PI’s best advice on how to successfully advance the science career is going to be colored in may cases by practicality of making reasonable and predictable forward progress. I recently remarked that the primary goal of a thesis-proposal committee is to say “gee that’s nice, now pick one quarter of what you’ve proposed and do that for your dissertation/defense“. Free range scientists often have much, much larger ideas than can fit into a single unit of productivity. This is going to almost inevitably mean the PI is reining in the “freedom” of the trainee. Also see: finite resources of the laboratory. Another common PI mentoring pinch point on “freedom” has to do with getting manuscripts actually published. The PI has tremendous experience in what it takes to get a paper into a particular journals. They may feel it necessary to push the trainee to do / not do specific experiments that will assist with this. They may feel it necessary to edit the hell out of the trainees’ soaring rhetoric which goes on for three times the length of the usual Intro or Discussion material. …..this does come at a cost to creativity and novelty.

If the “freedom” pays off, without undue cost to the trainee or PI or other lab members…fantastic! “Good mentoring!”

If that “freedom” does not pay off- grad student without a defendable project or publishable data, significant expenditure of laboratory resources wasted for no return – well this is “Bad mentoring!”

Different outcome means the quality of the same behaviors on the part of the PI is evaluated as polar opposites.

Asked and Answered

A tweet in response to a question I asked

I never thought about it when I was a starting postdoc.

But eventually I came to think that
a grad student's job was to learn how to answer questions, and a postdoc's job was to learn how to ask questions.

Artificial dichotomy, but seemed useful and reflected my trajectory.

— xaq (@xaqlab) June 2, 2021

said that perhaps a grad student’s job is to learn to answer questions and a postdoc’s job is to learn to ask questions.

I thought about that for half a second and concluded that this is backwards, for me. I think that I started into grad school thinking I knew how to ask scientific questions. I then spent the entirety of my time in grad school, right up until my committee signed off on my dissertation, learning the hard way that this was not so. I concluded that the main part of my graduate school training was learning how (not) to ask scientifically tractable questions.

In my postdoctoral training, I think that I learned how to answer questions. Not in the limited sense of “conduct this experiment, analyze the data and conclude a result”. Answering a question in the much broader sense of deploying available resources to address a scientifically tractable question and to bring this to an “answer” that was publishable in the scientific record.

I believe my career as a PI simply extends upon this, in the sense that my job is to secure the available resources in a broader sense and that “tractable” now includes the ability to direct the hands of more people. And the questions may no longer be my questions, but rather the questions of those who are in my laboratory. But it’s all just answering questions.