NHLBI gets serious about revision churning

Now we’re talkin’! The National Heart Lung and Blood Institute (NHLBI) is looking to take the final step necessary to break the cycle of unnecessary churning of grant revisions. As I have mentioned a time or two, I think that study sections have evolved into a position in which only revised grant applications are considered seriously. Furthermore I am convinced that this refusal to award top scores until the grant has been revised at least once is a big waste of time and effort in many cases. A “waste” because I dispute that the conduct of the eventual science is changed one bit by this process. The grant proposal itself, sure, a better document. But the resulting science? The investigator is (rightfully) going to pursue the science that makes the most sense to her, not whatever she has been forced to write by grantspersonship concerns.

I’ve also expressed my skepticism that the recent limitation of new grant proposals to a single round of revision is going to do much to overcome this study section tendency. Of course, I had a suggestion in a post on October 15, 2008:

The policy that I think would have been much better is one in which the NIH/CSR issues percentile ranks by revision status. That way ICs could simply apply a heavy bias for -01 applications over revisions to maintain whatever their target proportions of A0, A1 and A2 applications might be.

Apparently I am not the only one to reach this rather obvious conclusion; the NHLBI has published the following proposal which was considered at the October 21, 2008 meeting of the NHLBI Advisory Council:

Based on its analysis of recent NHLBI funding, the Committee concurred with the NIH decision to eliminate A2s. (Note: The Committee commenced its work prior to the NIH decision.) NHLBI funding data from the post-doubling era (FY 2004 – FY 2007) show that a large proportion of applications with an A0 percentile score fairly far down the percentile ranking were eventually funded, often as A1s or A2s. For example, about 80 percent of applications from established investigators (85 percent from new investigators) with an A0 score at the 20th percentile were eventually funded. The data also indicate a marked shift toward A2s and away from A0s. Furthermore, the data show substantial improvement in percentile scores between A0 applications and their A1s, and likewise between A1 applications and their A2s, for new investigators as well as established investigators. Based on these data, the Committee recommended eliminating A2s.

Next, the Committee considered percentiling A0s separately from A1s. FY 2007 data show this policy would have substantially lowered the percentile scores of A0s and substantially raised the scores of A1s. A retrospective model that re-percentiled NHLBI R01s and R21s by amendment status (and excluded A2s) showed that 267 resubmissions could have been avoided in FY 2007, 199 resubmissions avoided in FY 2006, and 31 resubmissions avoided in 2005.

Apparently the proposal was well received.

The Council enthusiastically supported the proposed policy.

[h/t: microfool]