Historical success rates predict current attitudes, redux

It is hard to overstate the problem that plummeting success rates at the NIH have caused for biomedical science careers. We have expectations for junior faculty that were developed in the 1980s and maybe into the 90s. Attitudes that are firmly entrenched in our senior faculty who got their first awards in the 1980s or even the 1970s…and then were poised to really rake it in during the doubling interval (since undoubled). Time for a trip down memory lane.

The red trace depicts success rates from 1962 to 2008 for R01 equivalents (R01, R23, R29, R37). These are not broken down by experienced/new investigators status, nor are new applications distinguished from competing continuation applications. The blue line shows total number of applications reviewed and the data in the 60s are listed as “estimated” success rates. (source)

The extension of these data into more recent FY can be found over at the RePORTER. I like to keep my old graph because NIH has this nasty tendency to disappear the good old days so we’ll forget about how bad things really are now. From 2011 to 2017 success rates hovered from 17 to 19% and in the past two years we’ve seen 21-22% success.

In the historical trends from about 1980 to the end of the doubling in 2002 we see that 30% success rates ruled the day as expected average. Deviations were viewed as disaster. In fact the doubling of the NIH budget over a decade was triggered by the success rates falling down into the 25% range and everyone screaming at Congress for help. For what it is worth, the greybeards when I was early career were still complaining about funding rates in the early 1980s. Was it because they were used to the 40% success years right before that dropping down to 30%? Likely. When they were telling us “it’s all cyclical, we’ve seen this before on a decade cycle” during the post-doubling declines….well it was good to see these sorts of data to head off the gaslighting, I can tell you.

Anyway, the point of the day is that folks who had a nice long run of 30% success rates (overall; it was higher once you were established, aka had landed one grant) are the ones who set, and are setting, current expectations. Today’s little exercise in cumulative probability of grant award had me thinking. What does this analysis look like in historical perspective?

I’m using the same 17.7% success rate for applications with white PIs reported in Hoppe et al and 30% as a sort of historical perspective number. Relevant to tenure expectations, we can see that the kids these days have to work harder. Back in the day, applicants had a 83.2% cumulative probability of award with just 5 applications submitted. Seems quaint doesn’t it? Nowadays a white PI would have to submit 9 applications to get to that same chance of funding.

How does that square with usual career advice? Well, of course newbs should not submit R01 in the first year. Get the lab up and running on startup, maybe get a paper, certainly get some solid preliminary data. Put the grant in October in year 2 (triaged), wait past a round to do a serious revision, put it in for July. Triaged again in October of Year 3. Two grants in, starting Year 3. Well now maybe things are clicking a bit so the PI manages to get two new proposals together for Oct and/or Feb and if the early submission gets in, another revision for July. So in Fall of Year 4 we’re looking at four or five submissions with a fairly good amount of effort and urgency. This could easily stretch into late Year 4.

Where do the kids these days fit in four more applications?

The cumulative probability of getting NIH grants

One of the career strategies we have discussed numerous times in various contexts is how many grant applications one should be submitting to the NIH. I have been a consistent advocate for …more. This is a recognition that success rates on a per-application basis have been below 20% for most of my career. Obviously this particular target number varies a lot. Sometimes we are talking about paylines, since that seems to be a hard target for success. Recent paylines from the NCI have been in the high single digits- 7-9%. Or we may talk about NIH-wide success rates overall, accounting for not just payline funding but pickups above the payline. These numbers change from year to year but mid to upper teens is a pretty fair general estimate.

My usual argument is that investigators who want to get funded should start with the assumption that they are no better than anyone else and need to base their strategy off the average success rate….at the very least.

Dumb old me, math challenged in the extreme, may have even expressed this as something like “If the payline is 10%, you need to put in 10 applications to even be in the game”. The more math savvy of you immediately chime in to correct me about calculating cumulative probabilities. This is not a difficult concept. I get it. But my mind likes to forget about it and I’ve never taken a blog whack at this issue directly, that I can recall.

Thanks to this handy binomial probability tool I googled up, we can now contemplate the abyss. Let us suppose a per-application success rate of 17.7%. Mid to upper teens, a decent place to start our discussion. And let us gate on the cumulative probability of at least one award. Well, if you put in 5 applications, your odds of one of them funding is 62.2% and if you put in 10 applications, this is 85.7%. Not too shabby. But it puts a very fine point that probabilities of award do not add. Fifteen applications are required to get to a 95% cumulative probability of at least one being awarded.

Reminder: We are not talking payline here. We are talking the 17.7% hit rate for the NIH wide average of everything that goes into awards including all of those various pickup behaviors. If you want the assurance of making the payline at a place like NCI, well…..Lord help you, amirite? That sounds totally rough and brutal and even unfair.

Now. Suppose that for some reason, say that your skin reflectance categorizes you as Black instead of white, your success rate was 10.7% instead of 17.7%.

Your chances are, of course, somewhat different. The cumulative advantage of putting in more grants, aka working harder, accrues less surely here as well. I’ve color coded a few ~equivalent cumulative probabilities for convenience. Using the Hoppe success rates, a Black applicant has to put in 9 proposals to get approximately the same 62% chance a white applicant achieves with only 5 applications. This same PI would have to put in 18 proposals to approximate the 86% hit rate the white PI gets with only 10 applications. About 25 proposals to get the 95% hit rate enjoyed by white applicants who put in 15 proposals.

Insert my favorite staring eyes emojii available on other platforms.

I would estimate that many Black folks, in the academy and elsewhere, are somewhat used to the idea that they need to grind a bit harder to achieve. At some level it probably doesn’t even really bother some on the day to day.

But this is a LOT harder. Putting in just one grant proposal is not easy. Particularly when you are a brand new Professor. But it is not trivial, even when you are a Full professor with some time and experience under your belt. [ Oh, btw, sidebar: Ginther et al 2018 has a little comment that I probably missed originally. “In results not reported, being a full professor increased the probability of NIH funding by 11.9 ppt (p < .001), but the funding gap remained -12.8 ppt (p < .001).” Yeah, age doesn’t help. ] When we are talking 5, 10, 25 applications it is maybe easy to overlook the sweat that goes into making new and credible proposals. Sure, some can be revised proposals and some are retreads of various prior proposals. But they take work and sweat to make them competitive. You are not going to enjoy the NIH-wide average hit rate with consistently below-average proposals!

This brings me back to a related issue that appeared in the Ginther et al 2011 report. “Applications from black and Asian investigators were significantly less likely to receive R01 funding compared with whites for grants submitted once or twice……black and Asian investigators are less likely to be awarded an R01 on the first or second attempt, blacks and Hispanics are less likely to resubmit a revised application, and black investigators that do resubmit have to do so more often to receive an award.“. I will admit I still don’t quite understand what they are presenting here at the end. It reads as though they are gating on Black investigators who do eventually win an award and do so on revision, not the A0 (this sample was back in the A2-permitted days, iirc). This whole passage, however, can be received as “well, just work a little harder to compensate” and to appear as if we’re only talking about an extra revision or two. I probably received this in this way myself on initially seeing the 2011 paper. And I have to say the “1.7 fold advantage” that is discussed in Hoppe for the per-application success rates comes in the same way. It can be received as, well, you just have to write two to get one. Because it focuses us on the “given you did get an award” instead of what it takes to get that award, statistically speaking.

But looking at these cumulative probability graphs really hits differently.

Black PIs don’t have to work just a little harder.

Black PIs have to work a LOT harder.