Racial Disparity in NIH Grants: Priority Scores

Unless you have been hiding under a rock, my NIH-focused Reader, you will have heard of the explosive findings of Ginther et al (2011) who reported on an analysis of racial and ethnic disparity in the review and funding of NIH grant applications.
There is a lot to discuss about these findings. A LOT. Well beyond the scope of one or even six blog posts. Commentary from the Office of Extramural Research, the NIMH and the Chronicle of Higher Education are worthwhile reads and there is a bit on National Public Radio as well. Blogger Bashir suggests* that these data prove that if you are African-American you have to be twice as good to succeed.
I’m going to jump right into some grant review geekery. I’m sure you are shocked.

These data are graphed from Table S1 and represented as percentages. I was interested in this Table, and motivated to re-graph the raw number, because of a curiosity about qualitative outcome. In my view the qualitative breakdown for scores is “likely fundable”, “could be picked up by Program” and “triaged”. These are moving targets across fiscal years, different study sections (the paper did not evaluate the percentile ranks used for funding decisions) and different Institutes or Centers of the NIH (which could have different score/pay relationships). Nevertheless, 100-150 maps reasonably well onto “fundable” and 151-200 onto the grey zone in which Program could possibly make a funding exception.
As a reminder, the old scoring system of the NIH (in place during the 2000-2006 interval used for the paper) went from 100 (best possible score) to 500 (worst possible score). Streamlining (“triage”) procedures that were in place meant that approximately half of the applications were not discussed at the study section meeting (based on the preliminary evaluation of the three assigned reviewers) and did not receive a panel-voted score (“Unscored” on the graph). In theory this meant that scores above about 250 should be rare. However, any proposal could be pulled up for discussion if any member of the panel wanted to do so, therefore there might be some initially poor-scoring proposals that received a correspondingly poor voted score. In addition, proposals that seemed initially promising might have flaws revealed during the discussion that drove their score down well past the putative line for initial streamlining.
Black applications are more likely to be triaged and less likely to be placed in the first two best-scoring bins. The disparity even continues into the “no way fundable” zone of 201-250. I’m interested in this because one hypothesis might be that when it comes to that last little push into the obviously-fundable territory, black applicants are not being favored. That might predict a boost in the just-missed-score bins. Not so. There really is a disproportionate triage burden here.
Supplementary Figure S1 provides pretty decent evidence that there is no disparity in funding outcome for a given priority score- so the decisions Program staff make to pick up a gray zone application (or, rarely, to pass over a highly scoring application) do not appear to play a role in the overall disparity effect.
I am also struck by this summary of the findings on grant submitting behavior:

On average, investigators had three to four Type 1 R01 grant applications each. We found that blacks and Asians resubmitted more times before being awarded an R01 (2.01, P < .06 and 1.85, P < 0.001, respectively) compared with whites (1.58), and at the same time blacks (45%) and Hispanics (56%) were significantly less likely to resubmit an unfunded application compared with white investigators (64%, P < 0.001) (table S6). We estimated Model 5 after introducing controls for the number of resubmissions and then estimated the model separately by the number of times a grant was submitted (table S7). Applications from black and Asian investigators were significantly less likely to receive R01 funding compared with whites for grants submitted once or twice. For grants submitted three or more times, we found no significant difference in award probability between blacks and whites; however, Asians remained almost 4 percentage points less likely to receive an R01 award (P < .05).
Together, these data indicate that black and Asian investigators are less likely to be awarded an R01 on the first or second attempt, blacks and Hispanics are less likely to resubmit a revised application, and black investigators that do resubmit have to do so more often to receive an award.

Emphasis added. You will recognize these as topics dear to my default advice to submit a lot of proposals, to revise and resubmit and generally to make friends with the process. I’ve occasionally had to smack down old school outdated advice to not revise triaged or even fairly poorly-scoring apps that were discussed.
You will also recall that at the same time I advise people to make use of the process as it stands, I complain that this default get-in-line-noob stuff puts a higher burden on the younger investigators. Maybe it places a similar burden on African-American applicants in some systematic way. Perhaps because of their job places and a lack of local support for spending a lot of time on a low-percentage behavior like grant submitting. Or maybe because there is a linear function of how much revising you have to do and your eventual learned-helplessness response of stopping swimming.
Perhaps this just points at the grantsmithing parts of how to effectively respond to criticism. Maybe African-American scientists are less likely to have good grant mentoring effectively available to them (no matter the type of employment location, remember they controlled for that).
Well, those are my thoughts for the day. This is a big issue that should be a big wake up to the NIH. I do hope this is not a mere flash in the pan that gets ignored. Likewise, I do hope we are not discussing the same disparity 5 or 10 years in the future and similarly wringing our hands.
__
Ginther, D., Schaffer, W., Schnell, J., Masimore, B., Liu, F., Haak, L., & Kington, R. (2011). Race, Ethnicity, and NIH Research Awards Science, 333 (6045), 1015-1019 DOI: 10.1126/science.1196783
*I hope the NIH is listening. If I am not mistaken this is coming from a postdoc who identifies on blog as African-American. Regardless of the outcome of upcoming analyses and pilot investigations mentioned by Tabak and Collins, there is going to be a HUGE perception problem. The NIH needs to take this just as seriously as resolving whatever obstacles and biases have resulted in the grant award disparity.