In hard times in NIH Grantlandia, guess who pays the steepest price?

A post over at Rock Talk blog describes some recent funding data from the NIH. The takeaway message is that every thing is down. Fewer grants awarded, fewer percentages of the applications being funded. Not exactly news to my audience. However, head over to the NIH data book for some interesting tidbits.

First up, my oldest soapbox, the new investigator. As you can see, up to FY2006 the PI who had not previously had any NIH funding faced a steeper hurdle to get a new grant (Type 1) funding compared to established investigators. This was despite the “New Investigator” checkbox at the top of the application and the fact that reviewers were instructed to give such applications a break. And they did in my experience….just not enough to actually get them funded. Study section discussion that ended with “…but this investigator is new and highly promising so that’s why I’m giving it such a good score…[insert clearly unfundable post-discussion score]” were not uncommon during my term of appointed service. So round about FY2007 the prior NIH Director, Zerhouni, put in place an affirmative action system to fund newly-transitioned independent investigators. There’s a great description in this Science news bit [PDF]. You can see the result below.

Interestingly, this will to maintain success rates of the inexperienced PIs at levels similar to the experienced PIs has evaporated for FY2011 and FY2013. See title.

Next, the slightly more subtle case of women PIs. This will be a two-grapher. First, the overall Research Project Grant success rate broken down by PI sex. As you can see, up through FY2002 there was a disparity which disappeared in the subsequent years. Miracle? Hell no. I guarantee you there has been some placing of the affirmative action fingers on the scale for the sex disparity as well. Fortunately, the elastic hasn’t snapped back in the past two FYs as it has for inexperienced investigators. But I’m keeping a suspicious eye on it, as should you. Notice how women trickle along juuuuust a little bit behind men? Interesting, isn’t it, how the disparity is never actually reversed? You know, because if whomever was previously advantaged even slipped back to disadvantaged (instead of merely equal) the whole world would end.

Moving along, we downshift to R01-equivalent grants so as to perform the analysis of new proposals versus competing continuation (aka, “renewal”) applications. There are mechanisms included in the “RPG” grouping that cannot be continued so this is necessary. What we find is that the disparity for woman PIs in continuing their R01/equivalent grants has been maintained all along. New grants have been level in recent years. There is a halfway decent bet that this may be down to the graybeard factor. This hypothesis depends on the idea that the longer a given R01 has been continued, the higher the success rate for each subsequent renewal. These data also show that a goodly amount of the sex disparity up through FY2002 was addressed at the renewal stage. Not all of it. But clearly gains were made. This kind of selectivity suggests the heavy hand of affirmative action quota filling to me.

This is why I am pro-quota and totally in support of the heavy hand of Program in redressing study section biases, btw. Over time, it is the one thing that helps. Awareness, upgrading women’s representation on study section (see the early 1970s)…New Investigator checkboxes and ESI initiatives* all fail. Quota-making works.

__
*In that Science bit I link it says:

Told about the quotas, study sections began “punishing the young investigators with bad scores,” says Zerhouni. That is, a previous slight gap in review scores for new grant applications from firsttime and seasoned investigators widened in 2007 and 2008, Berg says. It revealed a bias against new investigators, Zerhouni says.

CHE digs out some excuses from NIH as to why they are doing so little about Ginther

As you know I am distinctly unimpressed with the NIH’s response to the Ginther report which identified a disparity in the success rate of African-American PIs when submitting grant applications to the NIH.

The NIH response (i.e., where they have placed their hard money investment in change) has been to blame pipeline issues. The efforts are directed at getting more African-American trainees into the pipeline and, somehow, training them better. The subtext here is twofold.

First, it argues that the problem is that the existing African-American PIs submitting to the NIH just kinda suck. They are deserving of lower success rates! Clearly. Otherwise, the NIH would not be looking in the direction of getting new ones. Right? Right.

Second, it argues that there is no actual bias in the review of applications. Nothing to see here. No reason to ask about review bias or anything. No reason to ask whether the system needs to be revamped, right now, to lead to better outcome.

A journalist has been poking around a bit. The most interesting bits involve Collins’ and Tabak’s initial response to Ginther and the current feigned-helplessness tack that is being followed.

From Paul Basken in the Chronicle of Higher Education:

Regarding the possibility of bias in its own handling of grant applications, the NIH has taken some initial steps, including giving its top leaders bias-awareness training. But a project promised by the NIH’s director, Francis S. Collins, to directly test for bias in the agency’s grant-evaluation systems has stalled, with officials stymied by the legal and scientific challenges of crafting such an experiment.

“The design of the studies has proven to be difficult,” said Richard K. Nakamura, director of the Center for Scientific Review, the NIH division that handles incoming grant applications.

Hmmm. “difficult”, eh? Unlike making scientific advances, hey, that stuff is easy. This, however, just stumps us.

Dr. Collins, in his immediate response to the Ginther study, promised to conduct pilot experiments in which NIH grant-review panels were given identical applications, one using existing protocols and another in which any possible clue to the applicant’s race—such as name or academic institution—had been removed.

“The well-described and insidious possibility of unconscious bias must be assessed,” Dr. Collins and his deputy, Lawrence A. Tabak, wrote at the time.

Oh yes, I remember this editorial distinctly. It seemed very well-intentioned. Good optics. Did we forget that the head of the NIH is a political appointment with all that that entails? I didn’t.

The NIH, however, is still working on the problem, Mr. Nakamura said. It hopes to soon begin taking applications from researchers willing to carry out such a study of possible biases in NIH grant approvals, and the NIH also recently gave Molly Carnes, a professor of medicine, psychiatry, and industrial and systems engineering at the University of Wisconsin at Madison, a grant to conduct her own investigation of the matter, Mr. Nakamura said.

The legal challenges include a requirement that applicants get a full airing of their submission, he said. The scientific challenges include figuring out ways to get an unvarnished assessment from a review panel whose members traditionally expect to know anyone qualified in the field, he said.

What a freaking joke. Applicants have to get a full airing and will have to opt-in, eh? Funny, I don’t recall ever being asked to opt-in to any of the non-traditional review mechanisms that the CSR uses. These include phone-only reviews, video-conference reviews and online chat-room reviews. Heck, they don’t even so much as disclose that this is what happened to your application! So the idea that it is a “legal” hurdle that is solved by applicants volunteering for their little test is clearly bogus.

Second, the notion that a pilot study would prevent “full airing” is nonsense. I see very few alternatives other than taking the same pool of applications and putting them through regular review as the control condition and then trying to do a bias-decreasing review as the experimental condition. The NIH is perfectly free to use the normal, control review as the official review. See? No difference in the “full airing”.

I totally agree it will be scientifically difficult to try to set up PI blind review but hey, since we already have so many geniuses calling for blinded review anyway…this is well worth the effort.

But “blind” review is not the only way to go here. How’s about simply mixing up the review panels a bit? Bring in a panel that is heavy in precisely those individuals who have struggled with lower success rates- based on PI characteristics, University characteristics, training characteristics, etc. See if that changes anything. Take a “normal” panel and provide them with extensive instruction on the Ginther data. Etc. Use your imagination people, this is not hard.

Disappointingly, the CHE piece contains not one single bit of investigation into the real question of interest. Why is this any different from any other area of perceived disparity between interests and study section outcome at the NIH? From topic domain to PI characteristics (sex and relative age) to University characteristics (like aggregate NIH funding, geography, Congressional district, University type/rank, etc) the NIH is full willing to use Program prerogative to redress the imbalance. They do so by funding grants out of order and, sometimes, by setting up funding mechanisms that limit who can compete for the grants.

In the recent case of young/recently transitioned investigators they have trumpeted the disparity loudly, hamfistedly and brazenly “corrected” the study section disparity with special paylines and out of order pickups that amount to an affirmative action quota system [PDF].
All with exceptionally poor descriptions of exactly why they need to do so, save “we’re eating out seed corn” and similar platitudes. All without any attempt to address the root problem of why study sections return poorer scores for early stage investigators. All without proving bias, describing the nature of the bias and without clearly demonstrating the feared outcome of any such bias.

“Eating our seed corn” is a nice catch phrase but it is essentially meaningless. Especially when there are always more freshly trained PHD scientist eager and ready to step up. Why would we care if a generation is “lost” to science? The existing greybeards can always be replaced by whatever fresh faces are immediately available, after all. And there was very little crying about the “lost” GenerationX scientists, remember. Actually, none, outside of GenerationX itself.

The point being, the NIH did not wait for overwhelming proof of nefarious bias. They just acted very directly to put a quota system in place. Although, as we’ve seen in recent data this has slipped a bit in the past two Fiscal Years, the point remains.

Why, you might ask yourself, are they not doing the same in response to Ginther?