Comparing performance of within-payline and "select pay" pickup NIH grants at NIAID

October 26, 2011

Well, well, well. How timely. We were just discussing the situation in which some ICs of the NIH fund some subset of their grant applications out of the order of initial peer review. And what should I stumble upon (thanks to writedit) but some actual data which bear on the matter.

The NIAID website has an interesting analysis up that compares productivity measures for R01 grants from FY01-FY04. It divides the grants into those that were funded after receiving a score within their operating payline(s) and those that were funded via “Select Pay”. This is the term for out-of-order, exception funded proposals. Colloquially known as “pickups”.

NIAID describes the approach as:

Here’s how we conducted the study.

To measure productivity, we analyzed the number of publications from 2,104 applications that ranked within the payline (the WP cohort) and from 122 select pay applications (the SP cohort) shown in Figure 1.

For each indictor, we show only the middle 80 percent of the distribution (we removed the top and bottom 10 percent to make the figures easier to read). The horizontal line within each box represents the median.

Numbers for total publications, impact factor, and citations were 16,389, 102,786 and 196,117, respectively, for the WP cohort, and 860, 5,407 and 11,158 for the SP cohort.

Each indicator was scored for six years; for example, grants issued in FY 2002 were scored from FY 2002 to FY 2007.

Not entirely sure what they are graphing here, a typical box and whiskers plot would be 25%-75% described by the box. The whiskers, however, can be any number of descriptors. I guess the NIAID is putting the whiskers on the 20th and 80th percentiles…lot of room between 20% and 25% and between 75% and 80% if this is the case. [update: 10%ile and 90%ile of course; On reflection, I guess I should be less worried about the distance between 10% and 25% and between 75% and 90%.]

At any rate, the take home message is “no difference”. Same for Journal Impact Factor and number of actual citations of the papers.

So far as we can take such objective measures of grant productivity as relevant* to a fuzzier concept of “excellent science” or “impactful project”, this confirms what many of us familiar with grant review insist. Within that zone of payline and near-payline scores, there is no way to say the one grant is going to be much better than the other. Different, sure. But they are all going to be approximately as productive as each other, considering the groups as a whole.

Thus, the kvetching about how horrible it is that the NIH ICs fund some subset of their awards out of the order emerging from peer review is not really well justified. The “performance” of the NIH’s funded extramural research** is unlikely to be negatively affected by doing this.
__
*yes, I realize. But c’mon. Better something somewhat objective than continuing to shoot off our half-baked opinions without any evidence, no?

**extrapolating from NIAID’s data and with the same caveat about such measures of “performance”

Advertisements

No Responses Yet to “Comparing performance of within-payline and "select pay" pickup NIH grants at NIAID”


  1. The take-home message seems to me that the payline group has more outstandingly high numbers, although the median and lower ends are identical. My interpretation is that the very best (above the median of the payline group) stand out, while everyone else is essentically identical. I wonder whether there’s any correlation between that sub-group and impact scores; seems that’s where the real payline should be, and everything else might as well be pickups.

    Like

  2. drugmonkey Says:

    Do note the population size differences. 2,104 vs 122 grants.

    I wonder whether there’s any correlation between that sub-group and impact scores; seems that’s where the real payline should be, and everything else might as well be pickups.

    NIGMS did the analysis you are looking for, see this blog post. My read of those data is that “outstandingly high numbers” do not correlate with percentile rank of the grant either.

    Like

  3. SupplyDemand Says:

    The problems with science funding in the US are not because of a broken, unfair, or corrupt system that somehow misallocates research funds. The answer is much simpler: demand for grants vastly outstrips supply.
    And who controls factors that influence the demand for grants? Who keeps training more and more PIs that will eventually seek a grant and, thus, tighten the supply? The very same people that allege a “broken” grant system.

    Like

  4. whimple Says:

    The institutions are gatekeeping on who is defined as a PI for grantwriting purposes. There are too many PIs, which is driven by the rapacity of the University administrators for indirect costs associated with multi-grant PIs (institutions lose money on single-grant PIs).

    Like

  5. BugDoc Says:

    “institutions lose money on single-grant PIs”

    I think it’s really interesting/disturbing that there is a common perception out there that university basic research should be a revenue-generating operation. There are many comments in the blogosphere implying that universities are “subsidizing” NIH research. It seems to me that it is exactly the opposite – NIH is subsidizing university research for nothing more than the pleasure of being included in the Acknowledgements section of papers and talks. The universities get intellectual property rights, the universities reap the prestige associated with successful research faculty (ahem, UNIVERSITY employees) and they get F&A to help keep the lights on, which they should by rights have to pay for anyway. It’s crazy that we are perpetuating this idea that somehow universities have been the losers on this NIH bandwagon.

    Like

  6. Andrea Says:

    “this idea that somehow universities have been the losers on this NIH bandwagon”

    Buc dog

    NOBODY in her right mind and minimally informed would believe that universities have been the losers……..

    Ask Presidents, Deans and high end administrators… Ask families paying exorbitant tuition fees…. I would say that universities have been sucking money from NIH and everybody else to pay for their eccentricities.

    Like

  7. drugmonkey Says:

    If NIH funded research was a net loser for the Universities they would have dropped it long ago instead of doubling, tripling, nay septupling down on their bets…

    Like

  8. drugmonkey Says:

    and any time you see an accounting from a University that “proves” they lose money just remember the mantra…

    Money is fungible

    Like

  9. whimple Says:

    If NIH funded research was a net loser for the Universities they would have dropped it long ago instead of doubling, tripling, nay septupling down on their bets…

    Research didn’t used to be a net loser. Now Universities are doubling down because they don’t know how else to make up the lost revenue. I expect Universities to start dropping research any day now (or less drastically, to get rid of tenure).

    Like


  10. […] of the titans: fossil teeth show dinosaurs heading for the hills Comparing performance of within-payline and “select pay” pickup NIH grants at NIAID Chimpanzee consult (using genomics to help a baby chimp–you can’t defeat the […]

    Like

  11. AcademicLurker Says:

    If NIH funded research was a net loser for the Universities they would have dropped it long ago instead of doubling, tripling, nay septupling down on their bets…

    I never know quite what to make of the claim the research is a money loser precisely because if it is then the behavior of administrators makes no sense at all.

    Perhaps they are just irrational when it comes to research in the same way that they are irrational when it comes to big sports. The majority of even division 1 football programs are money losers, and yet universities just can’t resist the desire to play with the “big kids”.

    Like


  12. […] by using Programmatic Priority decision making to pick up awards out of the order of funding (which doesn't compromise science, btw). While I am generally in support of the system by which Program staff of NIH ICs decide to […]

    Like


  13. […] __ Additional Reading: Your Grant in Review: Productivity on Prior Awards Musing on NIGMS' grant performance data Another Look at Measuring the Scientific Output and Impact of NIGMS Grants Productivity Metrics and Peer Review Scores Mapping Publications to Grants Comparing performance of within-payline and "select pay" pickup NIH grants at NIAID […]

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: