Old Boys' Network Favors Men's Continuing Grants?
February 16, 2008
NIH has just released a whole slew of fascinating statistical data about grants awarded from 1998 through 2007. This is a 150+ slide Powerpoint deck with all kinds of grant award breakdowns catagorized by award type, awardee’s career stage, awardee’s sex, and all sorts of other interesting stuff. There is lots to discuss, but one of the first things to hit me when I skimmed the deck was one particular dramatic sex difference.
This graph breaks down the “success rate” over the last ten years for male and female applicants on “New” and “Continuing” R01 applications. R01’s are the basic research project grants that form the backbone of funding for virtually every academic biomedical research lab in the nation. They average about $350,000 per year, and are sufficient to fund a research program staffed with three or four scientists. The annual success rate is the number of grants awarded that fiscal year divided by the number of applications. New applications seek funding for a new research project for the first four or five years, while continuing applications seek funding for a succeeding four or five year period of a research project that is reaching the end of a four-five year new (or previous continuing) award. PIs are free to apply for consecutive continuing awards for as long as they please, and many R01 grants have been received continuous funding for decades.
One noteworthy feature of the data is that continuing applications fare much better than new ones, with roughly double the success rate over the entire ten years. Another one is that the success rate for continuing applications has declined much more than that issues to discuss, but I want to focus on something else right now.
Applications for new R01s from men and women PIs have had identical success rates over the last ten years. However, applications for continuing support with male PIs have had consistently higher success rates over this period, although the difference seems to have declined a bit over the last five years. This is of tremendous significance career-wise, as successfully obtaining a continuing award is frequently a sine qua non for obtaining tenure and/or promotion, and it is also of tremendous significance for the long-term viability of a PI’s laboratory regardless of career stage.
Why do the continuing R01 applications of men fare substantially better than those of women, while new applications do not? The review of a new application is based predominantly on the content of the application itself, while review of a continuing application also is heavily based on the extent to which the PI made good progress on achieving the goals of the preceding four-five year award.
Does the difference reflect operation of an explicit or implicit “old-boys’ network” that systematically overvalues the progress of male PIs compared women? Or does it reflect objectively poorer progress of women on preceding awards compared to men? If the latter, what might be the reasons women make poorer progress?
DrugMonkey 
February 16, 2008 at 5:41 pm
small (but important nit): the “success rate” is the number of grants funded divided by the number of grants that got scored. The NIH likes to play this little game where they increase the percentage of triaged grants to keep the “success rate” approximately constant with time in the face of declining inflation-adjusted budgets.
LikeLike
February 16, 2008 at 6:20 pm
I’m pretty sure this is wrong. This is the NIAID glossary entry:
http://www.niaid.nih.gov/ncn/glossary/default6.htm#s
LikeLike
February 16, 2008 at 7:18 pm
Your graph is very nicely displayed.
Although I don’t know what the answer is to this little quibble about how the data are added up, it is true that granting agencies benefit from keeping their success rates in a certain zone, and you can expect them to manipulate this. Sometimes it is less obvious than just picking a convenient way to count the grant applications. For instance, if an agency wants to lower the rate of funding (to not look like they give money to any kook that comes along) they can go around and drum up business, encouraging people to apply more, etc.
That sort of shenanigans makes analysis of the kind you are doing a little tricky.
LikeLike
February 16, 2008 at 8:08 pm
Interesting question, no idea about the answer. Could there be a seniority effect? E.g., the people applying for new ones are all younger and consequently more comparable, while the people applying for continued funding are more senior. Could there be an effect based on there being presumably more “very senior” males than “very senior” females? (Wild guess, because I know exactly nothing about the NIH funding scheme.)
I like the pink for the boys.
LikeLike
February 17, 2008 at 10:49 am
I suspect Schlupp is onto something critical in these data. If all continuations are lumped together there is going to be an increasing proportion of male PIs as you go upward in funding year. I would also hypothesize that success rates also go up as you go up in funding year since the momentum to continue increases. this, I’m just pulling out, however.
LikeLike
February 17, 2008 at 1:27 pm
I am shocked that you would do such a thing!
LikeLike
February 18, 2008 at 2:06 pm
Um … the sucess rate should ideally be independent of number for either gender. Even if there were only 100 women applying for continuing R01s versus 1000 men, those absolute numbers should not appear in the X% that get funded for either group. In other words, total number or proportion of genders provides no reason for 33% of women to be funded while 38% of men are funded.
LikeLike
February 18, 2008 at 2:21 pm
the point is that if seniority drives the effect it is not a direct consequence of gender. It may be an indirect one however.
LikeLike
February 18, 2008 at 4:23 pm
Even if there were only 100 women applying for continuing R01s versus 1000 men, those absolute numbers should not appear in the X% that get funded for either group.
I’m not sure if this or isn’t what DM was saying, but I understood Schlupp’s theory to be that the aggregate of men applying for continuing grants skews more senior than does the aggregate of women, making for higher success rates. For new grants, the aggregates are much more similar in seniority.
LikeLike
February 18, 2008 at 5:03 pm
JSinger, exactamundo. Now, I haven’t poured through the slides yet like PP. But going by previous experience with NIH power-point presentations it is often the case that the data are presented with some fairly common and very obvious alternative hypotheses left unexplicated. Annoying because you know they have the data and whatever point they are trying to make would be strengthened by presenting it.
One mystery here is the NIAID definition pulled by PhysioProf because my understanding of the most-common definition of “success” rate used NIH-wide allows for resubmitted-in-the-same-FY to be counted twice.
LikeLike
February 19, 2008 at 1:48 pm
Could this be an attribute of women leaving science between the time of obtaining the first grant and renewal, thus reducing the number of applications by women? This assumes that women leave academia more readily than men and if true just changes the question to “why do women leave more readily than men?”
LikeLike
February 19, 2008 at 2:59 pm
So, to further clear up what you’re saying: you’re arguing that for renewing applications, the male pool skews more senior, while the female pool tends to be younger? This I can believe. However, it implies that the very-senior scientists are more likely to be re-funded; while as I’ve heard it, the NIH is intentionally biased to younger/newer applicants.
Even so, the percentages should wash out the numbers or trends in distributions. Even if there are fewer very-senior women – they should ideally be funded at the same rate.
Lorax, the pipeline effect is definitely there. Women leave more readily than men for a number of reasons, but foremost among them is childbearing (and its cumulative side effect on gaps in one’s CV). However, given that the average age of first R01 is now mid-40’s, I wonder what else is going on … or whether it’s exactly the cumulative effect of gaps in one’s CV, on grant success.
LikeLike
February 19, 2008 at 3:12 pm
Are you kidding?
Assuming not, let us be emphatically clear. The “NIH” has a triple (at least) structure of review and prioritization of grants. The first is “peer review” at the point of the spear- the study section. This is the first and vastly most-important part since a very large bulk of run-of-the-mill funding is directly related to study section priority score.
The second most-important tier (actually several tiers) is the action of the Program staff of the Institutes and Centers. This is the place where the NIH can “reach down” to “pick up” grants out of order in the initial priority scores. They do so for an unending number of reasons having to do with scientific subareas of interest as well as to ensure diversity and other “workforce” goals. Which vary over time. For the past year or so, the NIH as a whole has been focused on the younger and transitioning scientist pool. So they have been doing a number of things to prioritize the funding of new scientists.
I take it this is where your “intentionally biased” comes from.
Two points, first that this is only necessary because study section review is already overwhelmingly biased against the younger investigator and in favor of the older investigators! Second point, is that the ameliorative behavior on the part of NIH is so far a drop in the ocean. IMO, of course. We can argue all day long about what represents “bias”, what represents legitimate and “fair” review of various types of applicant, etc. But the notion implied by your comment that things are all peachy for the young and dastardly poor for the aged researchers is not correct at present time.
LikeLike
February 19, 2008 at 3:46 pm
“Even so, the percentages should wash out the numbers or trends in distributions. Even if there are fewer very-senior women – they should ideally be funded at the same rate.”
Wash out a bit, yes, but not go away completely:
My point is that if the very senior people get mixed with the only slightly senior, then the ratio of ‘very senior’ to ‘slightly senior’ matters.
Let’s assume that every very senior person gets funded with a probability of 90%, and every slightly senior person with only 30% regardless of gender. Further, lets say we have 100 men and 100 women. Of the men, 90 are very senior, but only 10 of the women. Expected result: 0.9*90+0.3*10 = 84 men get funded, but only 0.9*10+0.3*90 = 36 women, even without any explicit gender bias. Obviously, my numbers are made up and not realistic at all, I’ve chosen them in order to illustrate the mechanism I had in mind.
LikeLike
February 19, 2008 at 4:17 pm
numerologist, you seem to be missing the point that the “competing continuation” data can include applications from about year 4 of the project (despite PPs comment it is fairly common for a completely new project to propose 3 yrs, not 4 or 5) all the way up to yr 50 or some such (writedit picked up one of these awhile ago). a few special cases of PI-swapping aside, the relationship of age-of-grant with age-of-PI is strong. a decrease in women as a percentage of PIs is also a function of PI age and therefore project duration.
to be clear, it is my hypothesis that competing continuations of longer duration do better than those of shorter tenure. given that that is true, we would need to use project duration as a factor to really determine the direct gender bias. if it is not true, i.e., competing continuations are uniformly successful across project durations, then you would be correct.
LikeLike
February 19, 2008 at 5:08 pm
Interesting data.
I wonder if reviewers are more likely to focus on deficiencies in women’s work and potentiality in men’s (as suggested in “Lifting a Ton of Feathers” by Paula Caplan). This might explain how the bias could play out (assuming it is a gender bias rather than a senority bias).
I do think that “senority bias” is actually quite confounded by gender as well. It’s not just that you get better at research as you move up the ranks, but that people may benefit from the reputation they have in the field. I’d be suprised if gender didn’t influence reputation. It’d be nice to blind some reviewers to the name of the applicant and see what happens.
I also wonder if the aforementioned mechanisms that exist to help out first time researchers are the root cause of these numbers. That is, somebody goes back and looks over the boarderline priority scores, trying to give people a fair shot, focusing on first time investigators. It may be that simply focusing on giving people a fair shot makes them stop and think, and that alone is enough to overcome the bias. It may also be someone at NIH is really trying to make sure women get funded their first time at the same rate as men (though I’ve never heard about this in particular being a priority).
LikeLike
February 19, 2008 at 5:25 pm
It may also be someone at NIH is really trying to make sure women get funded their first time at the same rate as men (though I’ve never heard about this in particular being a priority).
ooohh Becca, you bring up another complication. these “success” data place “funded grant” in the numerator, of course. And we know that grants get funded out of the rankings assigned out of initial peer review. Program is most certainly tasked with “representativeness” in various areas, starting with things like Congressional district, underfunded States or university “types”, scientific subarea and moving on to PI demographics. This process is fairly opaque. For our discussion today, it could very well be the case that Program diversity activities actually partially ameliorate the “real” bias, i.e., that at the study section level. A quick look at my go-to on initial scoring, the CSR FY 2004 databook, reveals no breakdown by PI gender.
LikeLike
February 20, 2008 at 4:21 pm
Here’s the word from the NIH on how exactly success rates are calculated, in response to this email inquiry: Are applications that are streamlined included in the success rate calculations?
Thank you for contacting DISHelp in the Reporting Branch of the Division of Information Services.
Success rate = grants funded / total applications. Applications with more than one amendment are counted once.
So streamlined grants still go in the denominator. I think this also means that you’re funded on the A2, the NIH considers you not to have ever written the -01 and -A1 grants. Likewise if you go three strikes you’re out with -01, -A1 and -A2, it only counts as one unfunded grant.
It appears that a better description of this calculation might be “grant project concept funding success rate”, rather than “grant funding success rate”. With success rates of 20% for new investigators, that means that 80% of their ideas suck. It’s just too bad it takes potentially 2.5 years and 75 pages of unfunded R01 applications to find out which ideas are in that 80%. It seems like even the seasoned veterans put in stupid (uh, “uncompetitive”) project concepts 2/3 of the time. 🙂
I wonder if there’s a difference between the new investigators and the seasoned veterans in terms of knowing when to bail on a research concept grant proposal. That is, are the seasoned pros better than the new investigators at getting back the pink sheets on the -01 and realizing this is never going to go anywhere and get on with a whole new proposal, or do they continue to grind out the failed grants until the NIH tells them they have to stop (at the -A2).
LikeLike
February 20, 2008 at 4:35 pm
whimple, the definition I’ve always heard is on this OER website:
“in the same fiscal year” being the critical concept. Thus it is likely only pairs of application/revisions (-01 / A1, A1 / A2) are going to be combined as “one” application. Furthermore, it is not all such pairs that will be treated this way if the PI skips an additional round or if it happens to fall across a FY line…
There is no doubt that this totally inexplicable definition of the denominator inflates the “success” rate relative to what it really should mean, i.e., each submission being considered independently. What is really bogus is that there is no way to determine the magnitude of this inflation. Ahh, the NIH in action…
LikeLike
February 20, 2008 at 5:22 pm
Yes, it’s very irritating. They clearly have the necessary raw data to answer all these questions. Why don’t they share it? My guess is that a suitably crafted FOIA request could in fact compel them to share it, were someone sufficiently motivated to make the request.
LikeLike
February 20, 2008 at 6:19 pm
I’m not so sure I have a problem with the -01, A1, A2 count NIH uses for success rates. This is the same manner, after all, that journals use to calculate their acceptance rates and I don’t have a problem with that. Moreover, who ever gets a ms accepted on the first shot (Nsc Letters aside). I suppose the bigger problem (which you have addressed many times before) is that the time frames of resubmission end up creating huge problems for keeping labs running and keeping people employed. My understanding is that NIH is working to get a solution to this resubmit time problem (and the election will also have an impact money wise, I hope). I thought the recommendations based on the solicitation for info were set to come out sometime soon… perhaps something concrete (hopefully not a block tied to a collective leg) will come of it?
LikeLike
February 20, 2008 at 6:50 pm
juniorprof, do keep in mind that my criticism of putting projects into the holding pattern of inevitable revision is combined with my assertion that the eventual conduct of the science is not even altered in most cases and is not substantially improved by this process.
LikeLike
February 20, 2008 at 8:06 pm
It might be useful for the NIH to do away with the -A2 application altogether. You get the comments back on the -01, you fix it, or you don’t.
LikeLike