Jocelyn Kaiser at ScienceInsider has obtained data on PI numbers from the NIH.

NIH PIs Graphic

Nice.

I think this graph should be pinned up right next to Sally Rockey’s desk. It is absolutely essential to any attempts to understand and fix grant application success rates and submission churning.

UPDATE 03/12/14: I should have noted that this graph depicts PIs who hold R01-equivalent grants (R01, R23, R29, R37 with ARRA excluded). The Science piece has this to say about the differential from RPG:

NIH shared these data for two sets of grants: research project grants (RPGs), which include all research grants, and R01 equivalents, a slightly smaller category that includes the bread-and-butter R01 grants that support most independent labs.

NIH-PIs-RPG-R01eqBut if you read carefully, they’ve posted the excel files for both the R01-equivalents and RPG datasets. Woo-hoo! Let’s get to graphing, shall we? There is nothing like a good comparison graph to make summary language a little more useful. Don’t you think? I know I do….

A “slightly smaller category” eh? Well, I spy some trends in this direct comparison. Let’s try another way to look at it. How about we express the difference between the number of RPG and R01-equivalent numbers to see how many folks have been supported on non-R01/equivalent Research Project Grants over the years…
NIHPI-RPGdifferentialWell I’ll be hornswaggled. All this invention of DP-this and RC-that and RL-whatsit and all the various U-mechs and P01 (Center components seem to be excluded) in recent years seemingly has had an effect. Sure, the number of R01 equivalent PIs only slightly drifted down from the end of the doubling until now (relieved briefly by the stimulus). So those in NIH land could say “Look, we’re not sacrificing R01s, our BreadNButter(TM) Mech!”. But in the context of the growth of nonR01 RPG projects, well….hmmm.

A communication to the blog raised an issue that is worth exploring in a little more depth. The questioner wanted to know if I knew why a NIH Program Announcement had disappeared.

The Program Announcement (PA) is the most general of the NIH Funding Opportunity Announcements (FOAs). It is described with these key features:

  • Identifies areas of increased priority and/or emphasis on particular funding mechanisms for a specific area of science
  • Usually accepted on standard receipt (postmarked) dates on an on-going basis
  • Remains active for three years from date of release unless the announcement indicates a specific expiration date or the NIH Institute/Center (I/C) inactivates sooner

In my parlance, the PA means “Hey, we’re interested in seeing some applications on topic X“….and that’s about it. Admittedly, the study section reviewers are supposed to conduct review in accordance with the interests of the PA. Each application has to be submitted under one of the FOAs that are active. Sometimes, this can be as general as the omnibus R01 solicitation. That’s pretty general. It could apply to any R01 submitted to any of the NIH Institutes or Centers (ICs). The PAs can offer a greater degree of topic specificity, of course. I recommend you go to the NIH Guide page and browse around. You should bookmark the current-week page and sign up for email alerts if you haven’t already. (Yes, even grad students should do this.) Sometimes you will find a PA that seems to fit your work exceptionally well and, of course, you should use it. Just don’t expect it to be a whole lot of help.

This brings us to the specific query that was sent to the blog, i.e., why did the PA DA-14-106 go missing, only a week or so after being posted?

Sometimes a PA expires and is either not replaced or you have happened across it in between expiration and re-issue of the next 3-year version. Those are the more-common reasons. I’d never seen one be pulled immediately after posting, however. But the NOT-DA-14-006 tells the tale:

This Notice is to inform the community that NIDA’s “Synthetic Psychoactive Drugs and Strategic Approaches to Counteract Their Deleterious Effects” Funding Opportunity Announcements (FOAs) (PA-14-104, PA-14-105, PA-14-106) have been reposted as PARs, to allow a Special Emphasis Panel to provide peer review of the applications. To make this change, NIDA has withdrawn PA-14-104, PA-14-105, PA-14-106, and has reposted these announcements as PAR-14-106, PAR-14-105, and PAR-14-104.

This brings us to the key difference between the PA and a PAR (or a PAS):

  • Special Types
    • PAR: A PA with special receipt, referral and/or review considerations, as described in the PAR announcement
    • PAS: A PA that includes specific set-aside funds as described in the PAS announcement

Applications submitted under a PA are going to be assigned to the usual Center for Scientific Review (CSR) panels and thrown in with all the other applications. This can mean that the special concerns of the PA do not really influence review. How so? Well, the NIDA has a generic-ish and long-running PA on the “Neuroscience Research on Drug Abuse“. This is really general. So general that several entire study sections of the CSR fit within it. Why bother reviewing in accordance with the PA when basically everything assigned to the section is, vaguely, in this sphere? And even on the more-specific ones (say, Sex-Differences in Drug Abuse or HIV/AIDS in Drug Abuse, that sort of thing) the general interest of the IC fades into the background. The panel is already more-or-less focused on those being important issues.  So the Significance evaluation on the part of the reviewers barely budges in response to a PA. I bet many reviewers don’t even bother to check the PA at all.

The PAR means, however, that the IC convenes their own Special Emphasis Panel specifically for that particular funding opportunity. So the review panel can be tailored to the announcement’s goals much in the way that a panel is tailored for a Request for Applications ( RFA) FOA. The panel can have very specific expertise for both the PAR and for the applications that are received and,  presumably, have reviewers with a more than average appreciation for the topic of the PAR. There is no existing empaneled population of reviewers to limit choices. There is no distraction from the need to get reviewers who can handle applications that are on topics different from the PAR in question. An SEP brings focus. The mere fact of a SEP also tends to keep the reviewer’s mind on the announcement’s goals. They don’t have to juggle the goals of PA vs PA vs PA as they would in  a general CSR panel.

As you know, Dear Reader, I have blogged about both synthetic cannabinoid drugs and the “bath salts” here on this blog now and again. So I can speculate a little bit about what happened here. These classes of recreational drugs hit the attention of regulatory authorities and scientists in the US around about 2009, and certainly by 2010. There have been a modest but growing number of papers published. I have attended several conference symposia themed around these drugs. And yet if you do some judicious searching on RePORTER you will find precious few grants dedicated to these compounds. It it no great leap of faith to figure that various PIs have been submitting grants on these topics and are not getting fundable scores. There are, of course, many possible reasons for this and some may have influenced NIDA’s thinking on this PA/PAR.

It may be the case that NIDA felt that reviewers simply did not know that they wanted to see some applications funded and were consequently not prioritizing the Significance of such applications. Or it may be that NIDA felt that their good PIs who would write competitive grants were not interested in the topics. Either way, a PA would appear to be sufficient encouragement.

The replacement of a PA with a PAR, however, suggests that NIDA has concluded that the problem lies with study section reviewers and  that a mere PA was not going to be sufficient* to focus minds.

As one general conclusion from this vignette, the PAR is substantially better than the PA when it comes to enhancing the chances for applications submitted to it. This holds in a case in which there is some doubt that the usual CSR study sections will find the goals to be Significant. The caveat is that when there is no such doubt, the PAR is worse because the applications on the topic will all be in direct competition with each other. The PAR essentially guarantees that some grants on the topic will be funded, but the PA potentially allows more of them to be funded.

It is only “essentially” because the PAR does not come with set-aside funds as does the RFA or the PAS. And I say “potentially” because this depends on their being many highly competitive applications which are distributed across several CSR sections for a PA.

__

*This is a direct validation of my position that the PA is a rather weak stimulus, btw.

As always when it comes to NIDA specifics, see Disclaimer.

NIH Multi-PI Grant Proposals.

February 24, 2014

In my limited experience, the creation, roll-out and review of Multi-PI direction of a single NIH grant has been the smoothest GoodThing to happen in NIH supported extramural research.

I find it barely draws mention in review and deduce that my fellow scientists agree with me that it is a very good idea, long past due.

Discuss.

In reflecting on the profound lack of association of grant percentile rank with the citations and quantity of the resulting papers, I am struck that it reinforces a point made by YHN about grant review.

I have never been a huge fan of the Approach criterion. Or, more accurately, how it is reviewed in practice. Review of the specific research plan can bog down in many areas. A review is often derailed off into critique of the applicant’s failure to appropriately consider all the alternatives, to engage in disagreement over the prediction of what can only be resolved empirically, to endless ticky-tack kvetching over buffer concentrations, to a desire for exacting specification of each and every control….. I am skeptical. I am skeptical that identifying these things plays any real role in the resulting science. First, because much of the criticism over the specifics of the approach vanish when you consider that the PI is a highly trained scientist who will work out the real science during the conduct of same. Like we all do. For anticipated and unanticipated problems that arise. Second, because there is much of this Approach review that is rightfully the domain of the peer review of scientific manuscripts.

I am particularly unimpressed by the shared delusion that the grant revision process by which the PI “responds appropriately” to the concerns of three reviewers alters the resulting science in a specific way either. Because of the above factors and because the grant is not a contract. The PI can feel free to change her application to meet reviewer comments and then, if funded, go on to do the science exactly how she proposed in the first place. Or, more likely, do the science as dictated by everything that occurs in the field in the years after the original study section critique was offered.

The Approach criterion score is the one that is most correlated with the eventual voted priority score, as we’ve seen in data offered up by the NIH in the past.

I would argue that a lot of the Approach criticism that I don’t like is an attempt to predict the future of the papers. To predict the impact and to predict the relative productivity. Criticism of the Approach often sounds to me like “This won’t be publishable unless they do X…..” or “this won’t be interpretable, unless they do Y instead….” or “nobody will cite this crap result unless they do this instead of that“.

It is a version of the deep motivator of review behavior. An unstated (or sometimes explicit) fear that the project described in the grant will fail, if the PI does not write different things in the application. The presumption is that if the PI does (or did) write the application a little bit differently in terms of the specific experiments and conditions, that all would be well.

So this also says that when Approach is given a congratulatory review, the panel members are predicting that the resulting papers will be of high impact…and plentiful.

The NHLBI data say this is utter nonsense.

Peer review of NIH grants is not good at predicting, within the historical fundable zone of about the top 35% of applications, the productivity and citation impact of the resulting science.

What the NHLBI data cannot address is a more subtle question. The peer review process decides which specific proposals get funded. Which subtopic domains, in what quantity, with which models and approaches… and there is no good way to assess the relative wisdom of this. For example, a grant on heroin may produce the same number of papers and citations as a grant on cocaine. A given program on cocaine using mouse models may produce approximately the same bibliometric outcome as one using humans. Yet the real world functional impact may be very different.

I don’t know how we could determine the “correct” balance but I think we can introspect that peer review can predict topic domain and the research models a lot better than it can predict citations and paper count. In my experience when a grant is on cocaine, the PI tends to spend most of her effort on cocaine, not heroin. When the grant is for human fMRI imaging, it is rare the PI pulls a switcheroo and works on fruit flies. These general research domain issues are a lot more predictable outcome than the impact of the resulting papers, in my estimation.

This leads to the inevitable conclusion that grant peer review should focus on the things that it can affect and not on the things that it cannot. Significance. Aka, “The Big Picture”. Peer review should wrestle over the relative merits of the overall topic domain, the research models and the general space of the experiments. It should de-emphasize the nitpicking of the experimental plan.

…or maybe it is.

One of the things that I try to emphasize in NIH grant writing strategy is to ensure you always submit a credible application. It is not that difficult to do.

You have to include all the basic components, not commit more than a few typographical errors and write in complete sentences. Justify the importance of the work. Put in a few pretty pictures and plenty of headers to create white space. Differentiate an Aim from a hypothesis from an Experiment.

Beyond that you are often constrained by the particulars of your situation and a specific proposal. So you are going to have to leave some glaring holes, now and again. This is okay! Maybe you are a noob and have little in the way of specific Preliminary Data. Or have a project which is, very naturally, a bit of a fishing expedition hypothesis generating, exploratory work. Perhaps the Innovation isn’t high or there is a long stretch to attach health relevance.

Very few grants I’ve read, including many that were funded, are even close to perfect. Even the highest scoring ones have aspects that could readily be criticized without anyone raising an eyebrow.

The thing is, you have to be able to look at your proposal dispassionately and see the holes. You should have a fair idea of where trouble may lie ahead and shore up the proposal as best you can.

No preliminary data? Then do a better job with the literature predictions and alternate considerations/pitfalls. Noob lab? Then write more methods and cite them more liberally. Low Innovation? Hammer down the Significance. Established investigator wanting to continue the same-old, same-old under new funding? Disguise that with an exciting hypothesis or newish-sounding Significance link. (Hint: testing the other person’s hypothesis with your approaches can go over great guns when you are in a major theoretical dogfight over years’ worth of papers.)

What you absolutely cannot do is to leave the reviewers with nothing. You cannot leave gaping holes all over the application. That, my friends, is what drops you* below the “credible” threshold.

Don’t do that. It really does not make you any friends on the study section panel.

__
*This is one case where the noob is clearly advantaged. Many reviewers make allowances for a new or young-ish laboratory. There is much less sympathy for someone who has been awarded several grants in the past when the current proposal looks like a slice of Swiss cheese.

Existing commitments will be honored but NOT-CA-14-023 makes it clear:

Effective immediately, no new nominations for NCI MERIT (R37) awards will be made.  In addition, NCI MERIT (R37) extensions will not be considered.

As a reminder, competing continuation R01s that score very well can be nominated for R37 which means that you get 10 years of non-competing instead of the usual limit to 5. That last bit in the quote refers to the fact that apparently these things can be extended even past the first 10 years.

 

A search on RePORTER shows that the NCI has about* 43 of these on the books at the moment.

 

*I didn’t screen for supplements or other dual entries.

This question is mostly for the more experienced of the PItariat in my audience. I’m curious as to whether you see your grant scores as being very similar over the long haul?

That is, do you believe that a given PI and research program is going to be mostly a “X %ile” grant proposer? Do your good ones always seem to be right around 15%ile? Or for that matter in the same relative position vis a vis the presumed payline at a given time?

Or do you move around? Sometimes getting 1-2%ile, sometimes midway to the payline, sometimes at the payline, etc?

This latter describes my funded grants better. A lot of relative score (i.e., percentile) diversity.

It strikes me today that this very experience may be what reinforces much of my belief about the random nature of grant review. Naturally, I think I put up more or less the same strength of proposal each time. And naturally, I think each and every one should be funded.

So I wonder how many people experience more similarity in their scores, particularly for their funded or near-miss applications. Are you *always* coming in right at the payline? Or are you *always* at X %ile?

In a way this goes to the question of whether certain types of grant applications are under greater stress when the paylines tighten. The hypothesis being that perhaps a certain type of proposal is never going to do better than about 15%ile. So in times past, no problem, these would be funded right along with the 1%ile AMAZING proposals. But in the current environment, a change in payline makes certain types of grants struggle more.

I don’t. I just don’t. I cannot in anyway understand scientists who are offended that they have to some up with some thin veneer of health-relevance to justify the grant award they are seeking. The H in NIH stands for “Health”. The mission statement reads:

NIH’s mission is to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability.

Yeah, sure, if you end at the seventh word, you can convince yourself that the NIH is about basic research. Maybe you get to continue on to the fifteenth. But this is a highly selective reading. I just don’t see where it is a burden to think for a minute or two about whether you are doing anything to address the second half of the statement.

After all, you are asking the taxpayers of the US to front you some serious cash. Millions of dollars for many of the PIs who are complaining about how hard it is to get basic research grants funded (BRAINI proponents, I’m looking at you). It really isn’t that much of an insult to ask you to pay something back on the matter of public health.

An RT Tweet from @betenoire1 was making the rounds of my Twitter feed today. It points to a Facebook polemic from a Leon Avery, Phd. (CV; RePORTER). He says that he is Leaving Science.

I have decided, after 40 years as a lab scientist and 24 years running my own lab, to shut it down and leave. I write this to explain why, for those of my friends and colleagues who’d like to know. The short answer is that I’m tired of being a professor.

Okay, no problem. No problem whatsoever. Dude was appointed in 1990 and has been working his tail off for 24 years at the NIH funded extramural grant game. He’s burned out. I get this.

I have never liked being a boss. My happiest years as a scientist were when I was a student and then a postdoc. I knew I wouldn’t like running a lab, and I didn’t like it. This has always been true.

My immediate plans are to go back to school and get a degree in Mathematics. This too has been a passion of mine ever since high-school sophomore Geometry, when I first learned what math is really about. And my love of it has increased in recent years as I have learned more. It will be tremendous fun to go back and learn those things that I didn’t have the time or the money to study as an undergrad.

GREAT! This is awesome. You do one thing until you tire of it and then, apparently, you have the ability to retire into a life of the mind. This is FANTASTIC!

So what’s the problem? Well, he can’t resist taking a few swipes at NIH funded extramural science, even as he admits he was never cut out for this PI stuff from the beginning. And after a long and easy gig (more on that below) he is distressed by the NIH funding situation. And feels like his way of doing science is under specific attack.

For many years NIH was interested in funding basic research as well as research aimed directly at curing diseases. With the tightening funding has come a focus on so-called “translational research”. Now when we apply for funding we have to explain what diseases our work is going to cure.

Ok, actually, this is the “truthy” part that is launching a thousand discussions of the “real problem” at NIH. So I’m going to address this part to make it very clear to his fans and back thumpers what we are talking about. On RePORTER (link above) we find that Dr Avery had one grant for 22 years. Awarded in April of 1991 and his CV lists 1990 as his first appointment. So within 15 mo (but likely 9 mo given typical academic start dates from about July through Sept) he had R01 support that he maintained through his career. In the final 5 years, he was awarded the R37 which means he has ten years of non-competing renewal. I see another R21 and one more R01. This latter was awarded on the A1. So as far as we can tell, Professor Avery never had to work too hard for his NIH grant funding. I mean sure, maybe he was putting in three grants a round for 20 years and never managed to land anything more than what I have reviewed. Somehow I doubt this. I bet his difficulties getting the necessary grant funding to run his laboratory were not all that steep compared to most of the rest of us plebes.

And actually, his Facebook post backs it up a tiny bit.

And I’ve been lucky that the world was willing to pay me to do it. Now it is hard for me to explain the diseases my work will cure. It feels like selling snake oil. I don’t want to do it any more.

I think the people enthusiastically passing along this Fb post of his maybe should focus on the key bits about his personal desires and tolerance for the job. Instead of turning this into yet another round of: “successful scientist bashes the NIH system now that finally, after all this time of a sweet, sweet ride s/he experiences a bare minimal taster of what the rest of us have faced our entire careers”.

Final note on the title: Dude, by all means. Anyone who has had a nice little run with NIH funding and is no longer entused….LEAVE. We’ll keep citing you, don’t worry. Leave the grants to those of us who still give a crap, though, eh?

UPDATE (comment from @boehninglab):

The R37/MERIT award is an interesting beast in NIH-land. It is typically (exclusively?) awarded upon a successful competing continuation (now called renewal) R01 application. Program then decides in some cases to extend the interval of non-competition for another 5 years*. This, my friends, is person-not-project based funding.

The R37 is a really good gig….if you can get it.

So, given that I’m blogging about award disparity this week….I took a look at the R37s currently on the books for one of my favorite ICs.

There are 25 of them.

The PIs include

1 transgender PI.
4 female PIs
0 East Asian / East Asian-American PIs (that I could detect)
3 South Asian / South Asian-American PIs (that I could detect)
0 SubSaharan African / African-American PIs (that I could detect)
0 Latino PIs (that I could detect)

hmmm, not that strong of a job. How about another of my favorite ICs?

23 awards (Interesting because this IC is half the size of the above-mentioned one)

12 female PIs.
0 East Asian / East Asian-American PIs (that I could detect)
1-2 South Asian / South Asian-American PIs (that I could detect)
0 SubSaharan African / African-American PIs (that I could detect)
3-4 Latino PIs (that I could detect)

way better on the sex distribution. Whether this number of R37s reflects more than average good-old-folks clubbery or the above represents less than average I don’t know. 25 at another large IC close to my interests. 95ish (I didn’t parse for supplements) at another. Only 45ish at NCI. Clearly a big range relative to IC size.

Both of these are doing really poorly on East Asian/ Asian-American and African-American PIs. The first is pretty pathetic on Latino PIs as well.

On the other hand, good old white guys with grey hair or receding hairlines are doing quite well in the R37 stakes.

How are your favorite ICs doing, Dear Reader?

__
*The way I hear it. I have heard rumor that these can go beyond a total of 10 years of R37 but I’m not sure on that.

The takeaway message from the report of Ginther and colleagues (2011) on Race, Ethnicity and NIH Research Awards can be summed up by this passage from the end of the article:

Applications from black and Asian investigators were significantly less likely to receive R01 funding compared with whites for grants submitted once or twice. For grants submitted three or more times, we found no significant difference in award probability between blacks and whites; however, Asians remained almost 4 percentage points less likely to receive an R01 award (P < .05). Together, these data indicate that black and Asian investigators are less likely to be awarded an R01 on the first or second attempt, blacks and Hispanics are less likely to resubmit a revised application, and black investigators that do resubmit have to do so more often to receive an award.

Recall that these data reflect applications received for Fiscal Years 2000 to 2006.

Interestingly, we were just discussing the most recent funding data from the NIH with a particular focus on the triaged applications. A comment on the Rock Talk blog of the OER at NIH was key.

I received a table of data covering A0 R01s received between FY 2010 and FY2012 (ARRA funds and solicited applications were excluded). Overall at NIH, 2.3% of new R01s that were “not scored” as A0s were funded as A1s (range at different ICs was 0.0% to 8.4%), and 8.7% of renewals that were unscored as A0s were funded as A1s (range 0.0% to 25.7%).

I noted the following for a key distinction between new and competing-continuation applications.

The mean and selected ICs I checked tell the same tale, i.e., that Type 2 apps have a much better shot at getting funded after triage on the A0. NIDA is actually pretty extreme from what I can tell- 2.8% versus 15.2%. So if there is a difference in the A1 resubmission rate for Type 1 and Type 2 (and I bet Type 2 apps that get triaged on A0 are much more likely to be amended and resubmitted) apps, the above analysis doesn’t move the relative disadvantage around all that much. However for NIAAA the Type 1 and Type 2 numbers are closer- 4.7% versus 9.8%. So for NIAAA supplicants, a halving of the resubmission rate for Type 1 might bring the odds for Type 1 and Type 2 much closer.

So look. If you were going to try to really screw over some category of investigators you would make sure they were more likely to be triaged and then make it really unlikely that a triaged application could be revised into the fundable range. You could stoke this by giving an extra boost to triaged applications that had already been funded for a prior interval….because your process has already screened your target population to decrease representation in the first place. It’s a feed-forward acceleration.

What else could you do? Oh yes. About those revisions, poorer chances on the first 1-2 attempts and the need for Asian and black PIs to submit more often to get funded. Hey I know, you could prevent everybody from submitting too many revised versions of the grant! That would provide another amplification of the screening procedure.

So yeah. The NIH halved the number of permitted revisions to previously unfunded applications for those submitted after January 25, 2009.

Think we’re ever going to see an extension of the Ginther analysis to applications submitted from FY2007 onward? I mean, we’re seeing evidence in this time of pronounced budgetary grimness that the NIH is slipping on its rather overt efforts to keep early stage investigator success rates similar to experienced investigators’ and to keep women’s success rates similar to mens’.

The odds are good that the plight of African-American and possibly even Asian/Asian-American applicants to the NIH has gotten even worse than it was for Fiscal Years 2000-2006.

NIH Blames the Victim

January 16, 2014

Just look at this text from RFA-RM-13-017:

The overarching goal of the Diversity Program Consortium is to enhance the diversity of well-trained biomedical research scientists who can successfully compete for NIH research funding and/or otherwise contribute to the NIH-funded workforce. The BUILD and NRMN initiatives are not intended to support replication or expansion of existing programs at applicant institutions (for example, simply increasing the number of participants in current NIH-funded research training or mentoring programs would not be responsive to this funding announcement).

The three forgoing major initiatives share one thing in common: Make the black PIs better in the future.

The disparity we’ve been talking about? That is clearly all the fault of the current black PIs….they just aren’t up to snuff.

Specifics? also revealing

 

Goals for the NRMN include the following:

  • Working with the Diversity Program Consortium to establish core competencies and hallmarks of success at each stage of biomedical research careers (i.e., undergraduate, graduate, postdoctoral, early career faculty).

  • Developing standards and metrics for effective face-to-face and online mentoring.

  • Connecting students, postdoctoral fellows, and faculty in the biomedical research workforce with experienced mentors, including those with NIH funding, both in person and through online networks.

  • Developing innovative strategies for mentoring and testing efficacy of these approaches.

  • Active outreach is expected to be required to draw mentees into the network who otherwise would have limited access to research mentors.

  • Developing innovative and novel methods to teach effective mentoring skills and providing training to individuals who participate as mentors in the NRMN.

  • Providing professional development activities (grant writing seminars, mock study sections, etc.) and biomedical research career “survival” strategies, and/or facilitating participation in existing development opportunities outside the NRMN.

  • Enhancing mentee access to information and perceptions about biomedical research careers and funding opportunities at the NIH and increasing understanding of the requirements and strategies for success in biomedical careers through mentorship.

  • Creating effective networking opportunities for students, postdoctoral fellows, and early career faculty from diverse backgrounds with the larger biomedical research community.

  • Enhancing ability of mentees to attain NIH funding.

To my eye, only one of these comes even slightly close to recognizing that there are biases in the NIH system that work unfairly against underrepresented PIs.

Jeremy Berg made a comment

If you look at the data in the Ginther report, the biggest difference for African-American applicants is the percentage of “not discussed” applications. For African-Americans, 691/1149 =60.0% of the applications were not discussed whereas for Whites, 23,437/58,124 =40% were not discussed (see supplementary material to the paper). The actual funding curves (funding probability as a function of priority score) are quite similar (Supplementary Figure S1). If applications are not discussed, program has very little ability to make a case for funding, even if this were to be deemed good policy.

that irritated me because it sounds like yet another version of the feigned-helpless response of the NIH on this topic. It also made me take a look at some numbers and bench race my proposal that the NIH should, right away, simply pick up enough applications from African American PIs to equalize success rates. Just as they have so clearly done, historically, for Early Stage Investigators and very likely done for woman PIs.

Here’s the S1 figure from Ginther et al, 2011:
Ginther-S1

[In the below analysis I am eyeballing the probabilities for illustration’s sake. If I’m off by a point or two this is immaterial to the the overall thrust of the argument.]

My knee jerk response to Berg’s comment is that there are plenty of African-American PI’s applications available for pickup. As in, far more than would be required to make up the aggregate success rate discrepancy (which was about 10% in award probability). So talking about the triage rate is a distraction (but see below for more on that).

There is a risk here of falling into the Privilege-Thinking, i.e. that we cannot possible countenance any redress of discrimination that, gasp, puts the previously underrepresented group above the well represented groups even by the smallest smidge. But looking at Supplementary Fig1 from Gither, and keeping in mind that the African American PI application number is only 2% of the White applications, we can figure out that a substantial effect on African American PI’s award probability would cause only an imperceptible change in that for White PI applications. And there’s an amazing sweetener….merit.

Looking at the award probability graph from S1 of Ginther, we note that there are some 15% of the African-American PI’s grants scoring in the 175 bin (old scoring method, youngsters) that were not funded. About 55-56% of all ethnic/racial category grants in the next higher (worse) scoring bin were funded. So if Program picks up more of the better scoring applications from African American PIs (175 bin) at the expense of the worse scoring applications of White PIs (200 bin), we have actually ENHANCED MERIT of the total population of funded grants. Right? Win/Win.

So if we were to follow my suggestion, what would be the relative impact? Well thanks to the 2% ratio of African-American to White PI apps, it works like this:

Take the 175 scoring bin in which about 88% of white PIs and 85% of AA PIs were successful. Take a round number of 1,000 apps in that scoring bin (for didactic purposes, also ignoring the other ethnicities) and you get a 980/20 White/African-AmericanPI ratio of apps. In that 175 bin we’d need 3 more African-American PI apps funded to get to 100%. In the next higher (worse) scoring bin (200 score), about 56% of White PI apps were funded. Taking three from this bin and awarding three more AA PI awards in the next better scoring bin would plunge the White PI award probability from 56% to 55.7%. Whoa, belt up cowboy.

Moving down the curve with the same logic, we find in the 200 score bin that there are about 9 AA PI applications needed to put the 200 score bin to 100%. Looking down to the next worse scoring bin (225) and pulling these 9 apps from white PIs we end up changing the award probability for these apps from 22% to ..wait for it….. 20.8%.

And so on.

(And actually, the percentage changes would be smaller in reality because there is typically not a flat distribution across these bins and there are very likely more applications in each worse-scoring bin compared to the next better-scoring bin. I assumed 1,000 in each bin for my example.)

Another way to look at this issue is to take Berg’s triage numbers from above. To move to 40% triage rate for the African-AmericanPI applications, we need to shift 20% (230 applications) into the discussed pile. This represents a whopping 0.4% of the White PI apps being shifted onto the triage pile to keep the numbers discussed the same.

These are entirely trivial numbers in terms of the “hit” to the chances of White PIs and yet you could easily equalize the success rate or award probability for African-American PIs.

It is even more astounding that this could be done by picking up African-American PI applications that scored better than the White PI applications that would go unfunded to make up the difference.

Tell me how this is not a no-brainer for the NIH?

A post over at Rock Talk blog describes some recent funding data from the NIH. The takeaway message is that every thing is down. Fewer grants awarded, fewer percentages of the applications being funded. Not exactly news to my audience. However, head over to the NIH data book for some interesting tidbits.

2013-FundingByCareerStageFirst up, my oldest soapbox, the new investigator. As you can see, up to FY2006 the PI who had not previously had any NIH funding faced a steeper hurdle to get a new grant (Type 1) funding compared to established investigators. This was despite the “New Investigator” checkbox at the top of the application and the fact that reviewers were instructed to give such applications a break. And they did in my experience….just not enough to actually get them funded. Study section discussion that ended with “…but this investigator is new and highly promising so that’s why I’m giving it such a good score…[insert clearly unfundable post-discussion score]” were not uncommon during my term of appointed service. So round about FY2007 the prior NIH Director, Zerhouni, put in place an affirmative action system to fund newly-transitioned independent investigators. There’s a great description in this Science news bit [PDF]. You can see the result below.

Interestingly, this will to maintain success rates of the inexperienced PIs at levels similar to the experienced PIs has evaporated for FY2011 and FY2013. See title.

2013-FundingBySexofPINext, the slightly more subtle case of women PIs. This will be a two-grapher. First, the overall Research Project Grant success rate broken down by PI sex. As you can see, up through FY2002 there was a disparity which disappeared in the subsequent years. Miracle? Hell no. I guarantee you there has been some placing of the affirmative action fingers on the scale for the sex disparity as well. Fortunately, the elastic hasn’t snapped back in the past two FYs as it has for inexperienced investigators. But I’m keeping a suspicious eye on it, as should you. Notice how women trickle along juuuuust a little bit behind men? Interesting, isn’t it, how the disparity is never actually reversed? You know, because if whomever was previously advantaged even slipped back to disadvantaged (instead of merely equal) the whole world would end.

2013-FundingBySexandTypeR01Moving along, we downshift to R01-equivalent grants so as to perform the analysis of new proposals versus competing continuation (aka, “renewal”) applications. There are mechanisms included in the “RPG” grouping that cannot be continued so this is necessary. What we find is that the disparity for woman PIs in continuing their R01/equivalent grants has been maintained all along. New grants have been level in recent years. There is a halfway decent bet that this may be down to the graybeard factor. This hypothesis depends on the idea that the longer a given R01 has been continued, the higher the success rate for each subsequent renewal. These data also show that a goodly amount of the sex disparity up through FY2002 was addressed at the renewal stage. Not all of it. But clearly gains were made. This kind of selectivity suggests the heavy hand of affirmative action quota filling to me.

This is why I am pro-quota and totally in support of the heavy hand of Program in redressing study section biases, btw. Over time, it is the one thing that helps. Awareness, upgrading women’s representation on study section (see the early 1970s)…New Investigator checkboxes and ESI initiatives* all fail. Quota-making works.

__
*In that Science bit I link it says:

Told about the quotas, study sections began “punishing the young investigators with bad scores,” says Zerhouni. That is, a previous slight gap in review scores for new grant applications from firsttime and seasoned investigators widened in 2007 and 2008, Berg says. It revealed a bias against new investigators, Zerhouni says.

As you know I am distinctly unimpressed with the NIH’s response to the Ginther report which identified a disparity in the success rate of African-American PIs when submitting grant applications to the NIH.

The NIH response (i.e., where they have placed their hard money investment in change) has been to blame pipeline issues. The efforts are directed at getting more African-American trainees into the pipeline and, somehow, training them better. The subtext here is twofold.

First, it argues that the problem is that the existing African-American PIs submitting to the NIH just kinda suck. They are deserving of lower success rates! Clearly. Otherwise, the NIH would not be looking in the direction of getting new ones. Right? Right.

Second, it argues that there is no actual bias in the review of applications. Nothing to see here. No reason to ask about review bias or anything. No reason to ask whether the system needs to be revamped, right now, to lead to better outcome.

A journalist has been poking around a bit. The most interesting bits involve Collins’ and Tabak’s initial response to Ginther and the current feigned-helplessness tack that is being followed.

From Paul Basken in the Chronicle of Higher Education:

Regarding the possibility of bias in its own handling of grant applications, the NIH has taken some initial steps, including giving its top leaders bias-awareness training. But a project promised by the NIH’s director, Francis S. Collins, to directly test for bias in the agency’s grant-evaluation systems has stalled, with officials stymied by the legal and scientific challenges of crafting such an experiment.

“The design of the studies has proven to be difficult,” said Richard K. Nakamura, director of the Center for Scientific Review, the NIH division that handles incoming grant applications.

Hmmm. “difficult”, eh? Unlike making scientific advances, hey, that stuff is easy. This, however, just stumps us.

Dr. Collins, in his immediate response to the Ginther study, promised to conduct pilot experiments in which NIH grant-review panels were given identical applications, one using existing protocols and another in which any possible clue to the applicant’s race—such as name or academic institution—had been removed.

“The well-described and insidious possibility of unconscious bias must be assessed,” Dr. Collins and his deputy, Lawrence A. Tabak, wrote at the time.

Oh yes, I remember this editorial distinctly. It seemed very well-intentioned. Good optics. Did we forget that the head of the NIH is a political appointment with all that that entails? I didn’t.

The NIH, however, is still working on the problem, Mr. Nakamura said. It hopes to soon begin taking applications from researchers willing to carry out such a study of possible biases in NIH grant approvals, and the NIH also recently gave Molly Carnes, a professor of medicine, psychiatry, and industrial and systems engineering at the University of Wisconsin at Madison, a grant to conduct her own investigation of the matter, Mr. Nakamura said.

The legal challenges include a requirement that applicants get a full airing of their submission, he said. The scientific challenges include figuring out ways to get an unvarnished assessment from a review panel whose members traditionally expect to know anyone qualified in the field, he said.

What a freaking joke. Applicants have to get a full airing and will have to opt-in, eh? Funny, I don’t recall ever being asked to opt-in to any of the non-traditional review mechanisms that the CSR uses. These include phone-only reviews, video-conference reviews and online chat-room reviews. Heck, they don’t even so much as disclose that this is what happened to your application! So the idea that it is a “legal” hurdle that is solved by applicants volunteering for their little test is clearly bogus.

Second, the notion that a pilot study would prevent “full airing” is nonsense. I see very few alternatives other than taking the same pool of applications and putting them through regular review as the control condition and then trying to do a bias-decreasing review as the experimental condition. The NIH is perfectly free to use the normal, control review as the official review. See? No difference in the “full airing”.

I totally agree it will be scientifically difficult to try to set up PI blind review but hey, since we already have so many geniuses calling for blinded review anyway…this is well worth the effort.

But “blind” review is not the only way to go here. How’s about simply mixing up the review panels a bit? Bring in a panel that is heavy in precisely those individuals who have struggled with lower success rates- based on PI characteristics, University characteristics, training characteristics, etc. See if that changes anything. Take a “normal” panel and provide them with extensive instruction on the Ginther data. Etc. Use your imagination people, this is not hard.

Disappointingly, the CHE piece contains not one single bit of investigation into the real question of interest. Why is this any different from any other area of perceived disparity between interests and study section outcome at the NIH? From topic domain to PI characteristics (sex and relative age) to University characteristics (like aggregate NIH funding, geography, Congressional district, University type/rank, etc) the NIH is full willing to use Program prerogative to redress the imbalance. They do so by funding grants out of order and, sometimes, by setting up funding mechanisms that limit who can compete for the grants.

2013-FundingByCareerStageIn the recent case of young/recently transitioned investigators they have trumpeted the disparity loudly, hamfistedly and brazenly “corrected” the study section disparity with special paylines and out of order pickups that amount to an affirmative action quota system [PDF].
All with exceptionally poor descriptions of exactly why they need to do so, save “we’re eating out seed corn” and similar platitudes. All without any attempt to address the root problem of why study sections return poorer scores for early stage investigators. All without proving bias, describing the nature of the bias and without clearly demonstrating the feared outcome of any such bias.

“Eating our seed corn” is a nice catch phrase but it is essentially meaningless. Especially when there are always more freshly trained PHD scientist eager and ready to step up. Why would we care if a generation is “lost” to science? The existing greybeards can always be replaced by whatever fresh faces are immediately available, after all. And there was very little crying about the “lost” GenerationX scientists, remember. Actually, none, outside of GenerationX itself.

The point being, the NIH did not wait for overwhelming proof of nefarious bias. They just acted very directly to put a quota system in place. Although, as we’ve seen in recent data this has slipped a bit in the past two Fiscal Years, the point remains.

Why, you might ask yourself, are they not doing the same in response to Ginther?