More on the new NIH policy on grant application revisions

In my brief original note on the new NIH policy limiting applicants to a single amendment (-01A1) of grants, I missed commenting on the accompanying press release. This latter overviews a little of the rationale for this step, starting with the problem

“Over the past several years, the number of applications submitted each year to NIH has doubled and the number of investigators applying for grants has increased by over 75 percent, increasing stress on the system, especially when confronted with stagnating budgets. This has led to scientists spending more time rewriting their applications and undue delays in the funding of outstanding projects,” Elias A. Zerhouni, M.D., NIH Director said.

and laying out the goal.

This new policy will help ensure earlier funding of high-quality applications and improve efficiencies in the peer review system.

Oh it will, will it?

The press release discusses the background that brought us to this point:

NIH analysis indicates that an increasing number of meritorious applicants that were ultimately funded had to resubmit their applications multiple times which increased burden on applicants and reviewers alike. … data reveals a reduction in the number of awards made to original applications. An increasing number of projects were funded only after one or more resubmissions. This trend has been increasing over recent years.

Right-o. A prior blog post overviewed my take on this dismal situation. I think my update at the end of that post is relevant:

UPDATE: PhysioProf supplies the long-term trends for all funded grants by revision status. Interesting to see how it developed over time. It reminds me of when the limitation for a maximum of two revisions of a given application were put into place in Oct 1996. One major argument was the relatively small number of applications that got funded as A2s and the further rarity of applications getting funded on additional revision. Assessing current trends by that logic should mean a return of A3 and A4 revisions, shouldn’t it?

As I mentioned before, the currently familiar limit to only two revised versions of grants was adopted in late 1996. This came at the end of a fairly obvious downward trend in the number and proportion of new grants funded without revision to the application. Looking at the new R01s listed in CRISP for the FY of 1995, I find 2591 new awards. There were 1391 (54%) funded unrevised, 749 (29%) as A1s, 314 (12%) as A2s, 112 (4.3%) as A3s, 19 (0.73%) as A4s, 5 (0.19%) as A5s and 1 (0.04%) as an A6. So some 94.7% of funded grants got there by the A2 revision stage in FY 1995. Thus, it made sense to cap off the number of permitted revisions because hardly anything was getting funded beyond A2.
The above figure also appears to show a beneficial effect of the A2-max policy on the number of grants funded at the unrevised stage. The proportion may have increased, however it is clear that the number of grants funded after the A1 and A2 revisions continued to increase as well. Was the supposedly beneficial effect a result of the policy change? or of the completely reversed NIH doubling interval?

Figure 1. NIH Appropriations (Adjusted for Inflation in Biomedical Research) from 1965 through 2007, the President’s Request for 2008, and Projected Historical Trends through 2010.

All values have been adjusted according to the Biomedical Research and Development Price Index on the basis of a standard set of relevant goods and services (with 1998 as the base year).* The trend line indicates average real annual growth between fiscal years 1971 and 1998 (3.34%), with projected growth (dashed line) at the same rate. The red square indicates the president’s proposed NIH budget for fiscal year 2008, also adjusted for inflation in biomedical research.

The NIH has a bunch of data slides here, including
-the proportion of R01s funded at A0-A2 revision stages as a function of the priority score for the A0 review.
-the proportion of A0-A2 awarded R01s from 1998-2007.
So I’m still at a loss. I don’t yet see where anything in the available data predicts that original, unrevised grant applications are going to have better success. Fit a trendline to the number of A0, A1 and A2 grants funded in the first figure, above. Look at the budget numbers in the second figure. The aberration in the early 2000s is associated with the budget doubling, not the change in policy with respect to amendments! In striking contrast to the 1996 change in policy, we have no data and can merely speculate about the fate of A3 or A4 revisions at present time, had there been no change back in 1996. I think that A3s would be a substantial part of the funded pool.
So what is going to happen after January 2009? Well, from a bean counter perspective, the funded A0s are going to increase. That’s because people who don’t get funded at the A1 stage are going to turn around the project in thinly veiled guise as their next “-01” application. Which will outcompete genuinely new -01 and -01A1 applications, on average. So the numbers may look better but the fundamental issues associated with delays to funding and wastes of the applicants’ and reviewers’ time will remain unchanged.
The reason is that this policy does nothing about the tendency of reviewers to focus on grantsmanship issues as an easy triage mechanism, instead of taking the “fish or cut bait” hard look at the genuinely new application the first time. The primary stage of review is the main driver here. The ameliorative measures should have accounted for the source of the problem and tried to address it more directly. The single amendment limit doesn’t do this.
One of the ways the original goal could be accomplished would be through reviewer education and instruction. Put the data figures in front of all reviewers and say “Bad dog! Stop deifying revision status and grantsmanship. Focus on the underlying science. What will really be accomplished through the review process- changes in the proposal only? or actual changes in the resulting science?” As we know, however, the CSR plays some funny games when it comes to providing reviewer instruction and guidance so this sort of thing isn’t going to happen…
The policy that I think would have been much better is one in which the NIH/CSR issues percentile ranks by revision status. That way ICs could simply apply a heavy bias for -01 applications over revisions to maintain whatever their target proportions of A0, A1 and A2 applications might be.
__
* NIH Office of Budget. Biomedical research and development price index (BRDPI). Bethesda, MD: National Institutes of Health, February 5, 2007. (Accessed August 16, 2007, at http://officeofbudget.od.nih.gov/UI/GDP_FromGenBudget.htm.)