Preferential Funding for First Submissions of NIH Grants

September 10, 2007

I’ve made reference a time or two to what I describe as “bias” for amended (revised) applications. In the lifecycle of the standard, investigator initiated research project grant (the R01) application, it is initially submitted and reviewed and if not funded, the application can be revised/amended one (called the A1) or two (A2) times. (Thereafter the PI must submit a substantially new proposal.) First, the evidence that revised applications score better and are more likely to get funded relative to initial submissions is readily available. The case is made first by reviewing the data provided in the CSR FY2004 databook (sadly, this is the only one available). If you look through Section III on Review Outcomes you can familiarize yourself with the respective and cumulative probability of getting funded at each submission stage. The case is perhaps more strongly made by performing a simple CRISP search on funded grants. For this latter, it helps to wildcard the Grant Number field on the first funded year of projects (e.g., “1R01%”) and perhaps gate by your favorite study section or funding Institute. (Gating by study section is preferred because if you don’t gate out catchall Special Emphasis Panel review panel designations you are contaminated by the RFAs and other situations in which at least some grants will be funded unrevised by default. Unfortunately one can’t screen by type of SEP, so as to parse RFAs, Panel Conflicts and the like.

From the study sections that would be the most usual suspects in my core and shell areas (i.e., the brain, behavior and pharmacology ones), this type of analysis shows that at present only some 10% or so of funded grants got through on first submission (if you do your own review, comment on the outcome, eh?). Interestingly if you go back in FY time, you will find that in recent history the A1’s were the biggest proportion of funded grants, followed by the unamended grants with A2’s forming a lesser fraction. Right around 2004 when things started looking grim, the number of A2’s funded started increasing while the unamended started declining. It is very difficult to conclude anything other than that this real-world outcome reflects the overall “problem”, i.e., increased submissions with lower/flatter NIH budgets. This is one of the stronger hints that there is nothing objective about the trend, i.e., that it is not about improving the science, see below. In rough terms, if we assume the much-bandied hard fund line of 10% (even though the eventual rate is a bit higher) this means that your application has to be in the top 1% to get funded unrevised. I think this is stupid and I think one good fix we could put in place would be to overtly bias in favor of un-amended applications. PhysioProf wants to know why.

First, let us review the problems with the revision process. It may help to assume for this exercise that we are talking about the pool of grants which will eventually be funded, but I think the general principle applies if you consider that the PI will one way or another generally be funded eventually. It takes a lot of effort from the PI and the lab members to revise a grant. From generating additional preliminary data, to writing the actual revision, etc. With personnel costs eating up a majority of grant funds, this is a big expenditure of NIH money. Despite the “rules” that grant preparation is not supposed to be done on NIH grant time, well, c’mon. Let’s be real. There is only so much time in a work year. Let’s take the 2080 hours in a standard work-year (yes, I know you all work 60hrs+ a week). A mere 40 hrs is almost 2% of this. That may be a bare minimum for a revision, never mind a fresh new proposal. So what’s 2%? A mere nothing, right? Except that you are putting out at least 3 of these a year and probably more. 5%? 10% of your time on revising grants? Did you ever look at what the big PIs are putting as their “effort” on grants? 10% or less as PI (gack!) and 5% or less as a major collaborating PI. Just sayin’.

There are efficiency issues. With the cycle for a given application taking 9 mo from submission to first-available-funding date a lot can happen in the lab in each cycle. Critical resources can disappear including well-trained techs, postdocs, Institutional commitments, human subject pools and expensive animal models such as genetic colonies or nonhuman primates. If these have to be spooled-up again, the essentially duplicative costs can be enormous. And what if the PI chooses to hold on to resources, underutilized, in anticipation? Well generally these costs are going to be borne in some way by…you guessed it, the PIs other NIH grant(s). So overall, from a business and taxpayer perspective, the NIH is burning a lot of money on grant revision.

Is it justified? The assumption in rebuttal to this position and the subtext for revising grants in the first place is that the process improves the grant application and, more critically the eventual conduct of the science. Right? The only legitimate purpose here is not to generate a better application but to actually select for better science. From my perspective, looking at fellow investigators’ grantsmanship and to some extent the grants my section reviews, the presumption that the revision process changes, never mind improves, the eventual research conduct is false in most cases.

One presumption is that the reviewers are catching fatal flaws in the research plan that would otherwise go undetected, wasting much time and NIH money. Pshaw. The PI is a trained scientist with a lot of experience in what it take to publish a paper in a given subfield, what it takes to meet the standards for a decent demonstration supporting hypotheses, what the appropriate “controls” are, etc. These applications are not the product of wet-behind-the-ears naifs, no, not even the New Investigator ones! So for the most part, the supposed flaws in the research plan boil down to empirical predictions (which can only be resolved in the doing), minor grantsmanship issues (it IS a limited application after all, not everything under the sun can be considered), methodological minutia and other crap. All science is limited and the beauty of investigator-initiated is that each PI works on what s/he finds interesting, not on what the reviewers find interesting. The ultimate arbiter of most of this is the paper review process anyway. Again, in my experience, the number of cases in which revision of an essentially in-play (i.e., not complete junk) application changes the eventual scientific conduct is low. “Improves” the eventual science? Lower.

Reviewers are not stupid so why has this emerged? Nobody likes it from the applicant standpoint. Well, this is a way to put applications in the holding pattern, waiting for funding. To eat away at the pool by sloughing off that fraction of PIs that just can’t sustain what it takes to go to the A2 stage (the CSR databook shows the New Investigator disadvantage here). There is some degree of the hazing mentality at play. And there are cyclical issues. The A2 of today needs to be prioritized to make up for the screwing it got at the -01 stage…because there were other A2s ahead of it “in line”. How to break the cycle?

How to come back from the brink by prioritizing funding unrevised applications? Well, one way or another it is going to hurt. Whenever a change is put in place there will necessarily be some applications (such as the above) being hosed. On the whole I think the benefits are worth it. Many more-established PIs have a mix of revised and unrevised grants under consideration at the same time anyway so for these it will be relatively painless. And the NIH could sneak up on it by gradually shifting the fund-unrevised targets from round to round.

7 Responses to “Preferential Funding for First Submissions of NIH Grants”

  1. PhysioProf Says:

    Your proposal is just rearranging deck chairs. The problem is that there are too many scientists competing for too little money. Period.

    Like

  2. drugmonkey Says:

    This is defeatist for one and a trifle short-sighted for another. Yes, in some senses EVERY possible proposal for change right now is trumped by the “too many scientists / too little money” concern. There are, however, systemic problems that are still present, even in the “good” days.
    I got my start in a pretty flush NIH period. It was by luck in many senses, there was no sense of a defined plan for transition (much my fault, not doubt) and even when I did get started I was up against all sorts of sources of bias against which I still rail. I argue that such sources disadvantaged the present and future conduct of science and the “problems” so to speak were only made more acute by the budget issues. The critique over what impact grant revision has still stands, even when A1 was the mode and -01 was close behind. Wasted taxpayer funds are wasted scientific opportunity no matter what the budget.

    In the practical, what I also fear and eagerly anticipate is that times of crises mean the opportunity for changes. The political will for upheaval of the NIH grant system is palpable this year. Naturally, I’d like the changes to go the way I think they should and not in the direction diametrically opposed. So it is worth generating discussion, light (and yes some heat) in the hope that something, anything, sticks.

    Like

  3. Concerned Scientist Says:

    When did the NIH become the welfare office for research science? With a budget larger than the GDP of most other countries, what is wrong with a little squeeze in the system to force PIs and institutions to find creative ways to find funding and also find creative ways to efficiently spend the money that they have.

    There should never be anything automatic in getting grants funded. Just because you have been in the business for 20 years doesn’t mean that you know what you are doing and should be funded just because of who you are. I think most agree that revised proposals are better than the original because of critism and time to develop the science. Lets fund the best science out there. Not all science deserves to be funded and not all PIs deserve to be funded.

    Sure it is difficult now. But PIs and the institutions really need to think hard about running their business. There is no guarantee that times will ever get any better, even with a Democrat in the White House.

    I just read your last comment and I completely agree. Now is the time to really put the state of NIH funded research under the microscope, from top to bottom. Direct/Indirect costs, postdoc training, graduate education. It is an old system that needs a change in the right direction.

    Like

  4. drugmonkey Says:

    “I think most agree that revised proposals are better than the original because of critcism and time to develop the science.”

    Really? Who are these “most”? Do you mean most applicants when it is their proposal that is being turned back with a 170 and no really substantive problems? Or is it that most reviewers agree that the proposal is better? I agree with the latter in fact, the proposals themselves are almost always better in revision. Tighter, more focused, better defended and outlined. Slicker. More impervious to the “usual” critiques. Etc.

    My point in this post is that this process fails to substantially affect, never mind improve, the actual conduct of the resulting science. So if this is what you mean, well, I dispute your contention, whether or not “most” agree.

    Keep in mind that I am not referring to a very bad, clearly triage-worthy POS app. I am referring to the really good ones in which the “bones” so to speak are readily apparent from the get go. As I said, it helps to think about the pool of “will eventually be funded by the A2” applications when thinking about my comments here.

    “Lets fund the best science out there. Not all science deserves to be funded and not all PIs deserve to be funded.”

    Do tell. How do you make these determinations?

    In case it is not obvious, this is a constant theme of mine. That it is damn hard to say what is “the best science”, particularly when you recognize the fact that “Health” is important in the NIH and “Institutes” is plural. It is not, repeat not, the National Institute of the BestPossibleBiomedicalScienceAsJudgedByCurrentHotTrends.

    Like

  5. whimple Says:

    How do you feel about the 270 that comes back with “we’d like to see some more preliminary data to demonstrate the feasibility of the project”?

    Like

  6. drugmonkey Says:

    Don’t get me started whimple! The Catch22 of preliminary data is a big problem. Not one that I have facile answers for though. It is one of these things where the principle sounds good but then it gets taken way too far.

    Sometimes reviews work better and sometimes worse, it can depend on field and subspecialty. I think I tend to get away lightly in this regard with my proposals. I am generally able to show the components of technical feasibility but rarely the substantive “feasibility”. It is rare that I have Aim I or even Experiment 1 mostly completed which is, I am imagining, the scenario you are raising and what I mean by substantive feasibility.

    On study section I’ve seen it argued just about every direction. People will go to bat for a minimal preliminary data app if they really like it. They will also rely on this StockCritique if they don’t like it and need shorthand. Sometimes it is clear to all that Preliminary Data are really important to the development of a solid research plan.

    So how do I feel about it? Well it is an easy out. A StockCritique. This doesn’t mean that it isn’t justified in some or even most cases. But I will say this. If you (one) get(s) a comment like this, you HAVE to give them something in the revision. It doesn’t have to be everything they seem to be “requesting”. But you need to give your advocates something to work with.

    Like

  7. Piled Higher, Deeper Says:

    By the way NIH sees it, though, you are not supposed to use Year 5 of the previous R01 to generate preliminary data for somewhat unrelated submissions, are you? So how is this supposed to be funded? R21s are the only thing for this purpose but these increasingly require Prelim Data too. This part of the system is farked and EUREKA is the admission of this.

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: