Racial Disparity in NIH Grants: Priority Scores

August 19, 2011

Unless you have been hiding under a rock, my NIH-focused Reader, you will have heard of the explosive findings of Ginther et al (2011) who reported on an analysis of racial and ethnic disparity in the review and funding of NIH grant applications.
ResearchBlogging.orgThere is a lot to discuss about these findings. A LOT. Well beyond the scope of one or even six blog posts. Commentary from the Office of Extramural Research, the NIMH and the Chronicle of Higher Education are worthwhile reads and there is a bit on National Public Radio as well. Blogger Bashir suggests* that these data prove that if you are African-American you have to be twice as good to succeed.
I’m going to jump right into some grant review geekery. I’m sure you are shocked.

Ginther11-PriorityScores.png


These data are graphed from Table S1 and represented as percentages. I was interested in this Table, and motivated to re-graph the raw number, because of a curiosity about qualitative outcome. In my view the qualitative breakdown for scores is “likely fundable”, “could be picked up by Program” and “triaged”. These are moving targets across fiscal years, different study sections (the paper did not evaluate the percentile ranks used for funding decisions) and different Institutes or Centers of the NIH (which could have different score/pay relationships). Nevertheless, 100-150 maps reasonably well onto “fundable” and 151-200 onto the grey zone in which Program could possibly make a funding exception.
As a reminder, the old scoring system of the NIH (in place during the 2000-2006 interval used for the paper) went from 100 (best possible score) to 500 (worst possible score). Streamlining (“triage”) procedures that were in place meant that approximately half of the applications were not discussed at the study section meeting (based on the preliminary evaluation of the three assigned reviewers) and did not receive a panel-voted score (“Unscored” on the graph). In theory this meant that scores above about 250 should be rare. However, any proposal could be pulled up for discussion if any member of the panel wanted to do so, therefore there might be some initially poor-scoring proposals that received a correspondingly poor voted score. In addition, proposals that seemed initially promising might have flaws revealed during the discussion that drove their score down well past the putative line for initial streamlining.
Black applications are more likely to be triaged and less likely to be placed in the first two best-scoring bins. The disparity even continues into the “no way fundable” zone of 201-250. I’m interested in this because one hypothesis might be that when it comes to that last little push into the obviously-fundable territory, black applicants are not being favored. That might predict a boost in the just-missed-score bins. Not so. There really is a disproportionate triage burden here.
Supplementary Figure S1 provides pretty decent evidence that there is no disparity in funding outcome for a given priority score- so the decisions Program staff make to pick up a gray zone application (or, rarely, to pass over a highly scoring application) do not appear to play a role in the overall disparity effect.
I am also struck by this summary of the findings on grant submitting behavior:

On average, investigators had three to four Type 1 R01 grant applications each. We found that blacks and Asians resubmitted more times before being awarded an R01 (2.01, P < .06 and 1.85, P < 0.001, respectively) compared with whites (1.58), and at the same time blacks (45%) and Hispanics (56%) were significantly less likely to resubmit an unfunded application compared with white investigators (64%, P < 0.001) (table S6). We estimated Model 5 after introducing controls for the number of resubmissions and then estimated the model separately by the number of times a grant was submitted (table S7). Applications from black and Asian investigators were significantly less likely to receive R01 funding compared with whites for grants submitted once or twice. For grants submitted three or more times, we found no significant difference in award probability between blacks and whites; however, Asians remained almost 4 percentage points less likely to receive an R01 award (P < .05).
Together, these data indicate that black and Asian investigators are less likely to be awarded an R01 on the first or second attempt, blacks and Hispanics are less likely to resubmit a revised application, and black investigators that do resubmit have to do so more often to receive an award.

Emphasis added. You will recognize these as topics dear to my default advice to submit a lot of proposals, to revise and resubmit and generally to make friends with the process. I’ve occasionally had to smack down old school outdated advice to not revise triaged or even fairly poorly-scoring apps that were discussed.
You will also recall that at the same time I advise people to make use of the process as it stands, I complain that this default get-in-line-noob stuff puts a higher burden on the younger investigators. Maybe it places a similar burden on African-American applicants in some systematic way. Perhaps because of their job places and a lack of local support for spending a lot of time on a low-percentage behavior like grant submitting. Or maybe because there is a linear function of how much revising you have to do and your eventual learned-helplessness response of stopping swimming.
Perhaps this just points at the grantsmithing parts of how to effectively respond to criticism. Maybe African-American scientists are less likely to have good grant mentoring effectively available to them (no matter the type of employment location, remember they controlled for that).
Well, those are my thoughts for the day. This is a big issue that should be a big wake up to the NIH. I do hope this is not a mere flash in the pan that gets ignored. Likewise, I do hope we are not discussing the same disparity 5 or 10 years in the future and similarly wringing our hands.
__
Ginther, D., Schaffer, W., Schnell, J., Masimore, B., Liu, F., Haak, L., & Kington, R. (2011). Race, Ethnicity, and NIH Research Awards Science, 333 (6045), 1015-1019 DOI: 10.1126/science.1196783
*I hope the NIH is listening. If I am not mistaken this is coming from a postdoc who identifies on blog as African-American. Regardless of the outcome of upcoming analyses and pilot investigations mentioned by Tabak and Collins, there is going to be a HUGE perception problem. The NIH needs to take this just as seriously as resolving whatever obstacles and biases have resulted in the grant award disparity.

Advertisements

18 Responses to “Racial Disparity in NIH Grants: Priority Scores”

  1. daedalus2u Says:

    I have a question. When proposals are revised and then awarded, does the research the PI is going to do actually change? In other words, does revising the proposal change the work that the PI does when funded? Or are the revisions mostly just window dressing?

    Like

  2. DrugMonkey Says:

    I am one that doubts that the revision process changes the resulting science in any discernible way. For the vast majority of proposals anyway, I’m sure there are always exceptions.

    Like

  3. Joe Says:

    In my field, all the African-American post-docs do much shorter post-docs (~2yrs) before moving on to faculty positions. Often the faculty positions are at traditionally black institutions. If you tried to control for training years for comparison with professors from other racial groups, you would be comparing the really-hot young guns with a much broader group of black professors.
    It is also possible that the old-boys network is part of the effect. Grant funding is often about trust, trying to figure out if you can trust the applicant to do (and be able to do) what they say they will do. So reviewers give better scores to people they trust, i.e., people they have met, heard speak, know the mentors of, etc. The numbers of African-American professors are way too low, and I don’t know to what extent they have been assimilated into the old-boys network.

    Like

  4. DrugMonkey Says:

    PP hasn’t convinced me. I think that “trust” stuff is wrong. Far more likely to fund pedestrian, same-old stuff than it is to let the highly productive established folks flourish. Here’s a hint- established folks can write good proposals too. It is not really some big problem to hold them to it.

    Like

  5. Mu Says:

    It might be a really dumb question, but does the NIH process involve some personal presentation? At least in my area of government funding process, mainly SBIR, there’s no way I know if a project up for review is coming from a [enter ethnicity of choice] or not.
    And this information is vital to distinguish between biased review or some disparity in proposal quality (which, if shown, might again be based on disparity in education but would not be an indictment of the selection process).

    Like

  6. DrugMonkey Says:

    It might be a really dumb question, but does the NIH process involve some personal presentation?
    No it does not. Save in a few cases of the larger mechanisms.
    mainly SBIR, there’s no way I know if a project up for review is coming from a [enter ethnicity of choice] or not.
    In NIH review (NIH has SBIR too, you know), the Principal Investigator is named and is indeed one of the focii of the review itself… “Investigator” is one of 5 main criteria.
    Often times, that person is known to the reviewers through professional interactions- meetings and the like. remember the “underrepresented” part? It is not too much of a stretch to think the average black scientist is better known/recognized *precisely* because they are a rarity. I would assert this is the case in my field, for example.
    If the PI is not known to the reviewer, a quick Google search very often turns up a picture of the person on their University website. There may even be some clues in the “awards and recognitions” section of the biosketch.

    Like

  7. sis orguleri Says:

    thank For you science blogs. ismek istanbul şişörgü kursu,modelelerini öğrettiği eğitimlerinene başlıyor…
    In my field, all the African-American post-docs do much shorter post-docs before moving on to faculty positions. Often the faculty positions are at traditionally black institutions. If you tried to control for training years for comparison with professors from other racial groups, you would be comparing the really-hot young guns with a much broader group of black professors.

    Like

  8. Grad with(out) Braids Says:

    remember the “underrepresented” part? It is not too much of a stretch to think the average black scientist is better known/recognized *precisely* because they are a rarity. I would assert this is the case in my field, for example.
    As a black woman, this is definitely true. Being generally terrible with faces, it was a shock to go to my first conference and realize that everyone remembered me after the first day, and that many people recognized my face from being in the audience at a colloquium talk. I’ve even had black colleagues-of-colleagues reach out to me through email- and all this is without having a first authorship paper yet. My name’s as generic as Jane Doe, and yet I’m sure that when the time comes (hopefully!) to submit grants, people will know who I am.

    Like

  9. Eurotrip Says:

    So if it’s not possible to know the “race” (can’t believe you Americans still actually use the word race in 2011) of the applicants, then how can there be direct ethnic discrimination? Or is it easy to distinguish between White American and Black American names? Obviously an Asian or Hispanic name is quite easy to spot.

    Like

  10. Josephine Says:

    I looked through the paper, and I don’t think they controlled for the subfield that the researcher was in, or specifically what kind of training they had. I know that at my undergrad, Black students were EXTREMELY underrepresented in Statistics and engineering majors. Do scientists with that type of training have better luck getting grants? I have seen that having a rigorous quantitative background can often be an asset for Biomedical researchers. Also the thing about the Post-Docs also rang true for me, though they say that they controlled for “education and NIH training” but that may not include length of time spent in a Post-Doc position.

    Like

  11. whimple Says:

    Rockey’s comment is illuminating:
    As someone who has been involved in designing and implementing government peer review programs for over 25 years and one who believes strongly in peer review as the best way to identify good science, I find these results troubling as well.
    The fiction is that peer review identifies good science when actually peer review identifies science (and scientists) the peers like. OER should start their introspection by acknowledging this reality.

    Like

  12. DrugMonkey Says:

    Anecdotally from my grant reviewing experiences I see no evidence that the type of training department makes any difference in grant writing ability or assessment of the PI. (on R grants, fellowship review does focus on the transcript). Things that influence the review are just too far away from whether the PI trained in Psych, Cog Sci, comp Sci, pharm, bio, etc.

    Like

  13. daedalus2u Says:

    DrugMonkey, so what you are saying is that proposal review, rejection, revision, resubmission and ultimate acceptance is essentially theater, that it doesn’t change the outcome, the research that is done.
    I hope that everyone appreciates that funding science in this way doesn’t result in the “best science” being done, it results in the most popular science being done. If science funding lurches from one fad to another, non-faddish science won’t get done.
    Who decides what the next science fad should be? If it is press releases and MSM putting out puff pieces on “the next big thing”, then no amount of “peer review” is going to substitute good science for faddish science.

    Like

  14. DrugMonkey Says:

    You are incorrect in yOur assertion daedalus. The system is imperfect, true, but there is a diversity in what is judged meritorious enough for funding. Some fads are the “best science” at the time. Some “boring” stuff keeps pLugging away. Some amazing novelty is

    Like

  15. Eli Rabett Says:

    One possibility is that the program officers do not recommend resubmission symmetrically.

    Like

  16. DrugMonkey Says:

    What the stones program Officer said anything other than “I advise you to revise and resubmit” from 00-06???

    Like

  17. DrugMonkey Says:

    Related post from some RedSox fan:
    My response in the event I get moderated

    Your comment is awaiting moderation.
    Interesting analysis. Particularly since you choose to dismiss the triage as if it is nothing- to paraphrase a feminist mantra, equality will arise when the black PIs have only to do as well as the least accomplished white PIs.
    You ignore the fact that black PIs had to resubmit one more time, on average, to reach these stellar scores.
    You ignore the grey zone of funding exceptions (from this rough analysis, the second bin 151-200 is the closest we can get to this qualitative bin)
    So when you look at the very top of the most excellent proposals, perhaps your analysis stands. But then what? A sharp drop off for the run of the mill “good” black PIs?

    Like

  18. Eli Rabett Says:

    They get good at hinting, at least in NSF

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: