The Program Staff of the NIH decide what to fund, all else is advisory

October 25, 2011

…and if I have it right, the Director of a given IC is really the one who makes the decision to fund or not fund a given grant proposal…and even the input of their own Program Staff is only advisory.

Some comment over at writedit’s thread on paylines is incensed about the NCI:

how are the NCI officials better qualified then the peer reviewers to judge the importance of these “selected” studies?

Because They. Have. A. Different. Perspective. on the fields of science. A broader perspective. In the best of all worlds, the Program Staff have the opportunity to step back and look at the larger trends within the science that is within their jurisdiction. To see where current fads have left holes in their portfolio. To perhaps take a risk where the peers are conservative. To identify the duplications and overlaps within their own portfolio and adjust accordingly. To worry about the next generation of scientists. Etc.

How can individual scientists, or even a panel of 25-30 of them, possibly take the long view? They cannot. So there is a role for Program. We can debate whether a funding agency should be sensitive to the long view, balance of effort, inherent self-referential conservatism that emerges in science now and again, etc. I’ll come down squarely in favor of breadth on that one. But let us not pretend they have no functional role.

They should not be make funding decisions. They are messing up the once well established system.

Yes, yes they should be making funding decisions. This is the job of Program, actually. They are part and parcel of the “well established system”.

I’ve touched on this ever so briefly before, see Program Interferes with…. and NIH Administrators Ignore…

Oh glory…since I started writing this post, the commenter doubled down:

If the grants do not fit the NCI portofolio they should not be sent forward for peer review in the first instance; and more importantly these same grants that do not fit the portofolio in the first place should not be sent back for an A1 revision which consumes a whole lot of funds to generate additional preliminary data. I think this suggests that not all the decisions are made by well qualified individuals at NCI.

and an echo from yet another commenter….

If NCI had told me that my app. did not fit their portofolio, I wouldn’t have wasted freaking 4 yrs submitting A0 and A1( plus wasting peer review efforts). And before submitting my A0, I had even consulted my SRO. I’m pissed that NCI changed rules in the middle of the game

HAHAHAHAAHA!!!! This commenter no doubt would be raising a big stink if his/her grant got rejected without even going out to peer review. But the underlying principle that Program should be highly pro-active about refusing grants before they even get reviewed is stupid. There is no chance for reviewers who are more expert in the science to point out whether it fits the mission or not. Program staff are not omniscient. They need their extramural scientists to educate them. Not to dictate their job priorities, not at all. To educate. To provide a portion of the knowledge, information and evidence that Program staff require to do their part of the job. This is at the root of the investigator-initiated science funding system is it not? It is our job to make our case. The job of peer review to provide one viewpoint on that case. The job of Program Staff to provide another set of inputs.

I like this. I like the ability to make my case for what I am interested in studying. I would be far less enthralled to have to always fit into some pre-existing set of Programmatic interests. I think our friends over at writedit’s blog would be similarly distressed if there were a more heavy handed triage of applications prior to review.

The only way that would work is if someone at the Program level does a lot of triaging on the basis of the Abstract. Because I guarantee you the POs are not going to be reading all the apps in detail under such a scenario. Too many applications and too few POs. And if there were enough of them to do so? We’d all be crying foul about why it took 10X as many Program staff as they have now! Think of how many grants that would suck up. And they still couldn’t offer the kind of specific expertise that the current peer-review system can muster.

and back to the original commenter…..

I would scream bloody murder if the grant was peer reviewed, fell in the upper 10th percentile and then was told that it did not fit the portofolio…….to an extent that is what is happening at the NCI….but you are entitled to your opinion

Of course the grey zone has long been familiar to those of us who seek funding from some of the other institutes. I have my doubts, of course, about assertions that some of the ICs have or had a policy of sticking strictly to the outcome of the initial peer review. But….perhaps this has indeed been the case at NCI. And the Chicken Littles can be excused, a trifle, for not knowing how it goes down. We have only the hard data from NIGMS but these graphs fit very well with my subjective experiences as an applicant, a friend and colleague of applicants, watching what emerges from a study section on which I served, talking with POs, etc. Take this FY2010 R01 funding outcome graph from NIGMS as an example:

Open rectangles depict the number of grants reviewed, the dark bars the number funded. The X axis depicts percentile scores that emerged from initial peer review. What this shows is that genuine “skips” are relatively rare. There is a clear payline (formally published or not, you can see where it lies) below which almost everything gets funded. Above that line is the grey area. A zone above the payline (I note for those who are thinking that they “deserve” to get funded) in which only a subset of the grants get funded. Notice the trendline though? Even this is influenced strongly by the priority score/percentile rank, right? The chances of an application being funded as an exception to the payline increases as the score moves closer to that payline.

The NCI has apparently been talking a payline in the 7-8%ile zone and mollifying investigators that they will “consider” everything up to some 25 %ile for exception funding. Pickups. So the scenario raised by the commenter is asking about apps in the 8-10%ile range and their chances of being skippped over. Unless NCI takes a very different approach to exception funding, the chances for such a score are probably still quite good. Once you get a couple of bins away from the payline in the NIGMS data above (and check the sidebar at the NIGMS page for prior Fiscal Year trends) then the chances for any application funding get pretty slim.

It is difficult for me to understand how anyone can look at the distribution of funded/unfunded grant application in this grey area and think that they “deserve” to be funded with scores that are above the payline. I say rather that people should feel lucky to get the nod, given that few of the apps are being funded.

Advertisements

No Responses Yet to “The Program Staff of the NIH decide what to fund, all else is advisory”


  1. You just don’t get it, holmes. Whatever was the reason my most recent application didn’t get funded is the exact particular reason that THE SYSTEM IS TOTALLY BROKEN. Last cycle, I got triaged, so PEER REVIEW IS BROKEN!!!!. This cycle, I got a great score but my application didn’t get funded, so HOW DARE PROGRAM STAFF DISREGARD PEER REVIEW!?!?!?

    Like

  2. whimple Says:

    Presumably the complaint comes from those who would be funded if a hard payline were established with no exceptions (guessing NCI: 12%), but who are not funded because they are above the more stringent “harder payline” (NCI:7%) and are not picked up programmatically (NCI:8-25%).

    Like

  3. DrugMonkey Says:

    Presumably the complaint comes from those who would be funded if a hard payline were established…

    Right. And next cycle these same exact people would be complaining about how three virtually identical me-too applications were funded while their amazingly innovative project went unfunded at 15%ile..

    Like


  4. You fuckes just don’t get it. The point is that my brilliant science must be funded, and if it isn’t, then the SYSTEM IS BROKEN AND MUST BE REFORMED.

    Like

  5. PI Says:

    The system is indeed broken. That is because demand for grants greatly outstrips supply, and that, in turn, is because you PIs keep training future PIs like there is no tomorrow, thereby increasing the demand for grants.
    I know… grad students are cheap. But remember that the chickens will eventually (in about 10 years) come home to roost. So, in essence, you have shot yourselves in the foot. Tight budgets have exacerbated the situation, but this crisis is largely your creation.

    Like

  6. arrzey Says:

    So who has a better system other than “my grant must be funded”? Having been inside & outside, there are mistakes being made. But I can’t think of a better way to do it that will not be more arbitrary and less fair, even to CPP.

    BTW, a young colleague (not in my lab) is in that gray area and just got her first R03 funded. I had reviewed the grant, as honestly as I could, and knew that it might end up being gray, but if it did bounce back, I wanted her to have some good feedback. But Staff decided to fund her. For every complainer, there are still ones that come out right.

    Like

  7. Drugmonkey Says:

    Funny how nobody uses their own funded-out-of-order grant as evidence of the broken system, incompetent review and/or factual errors, eh arrzey?

    Like

  8. ABCD Says:

    …It is difficult for me to understand how anyone can look at the distribution of funded/unfunded grant application in this grey area and think that they “deserve” to be funded with scores that are above the payline….

    The grey zone got increased under Varmus ( 8%tile points for established, 15 points for ESI) which is unprecedented. The gray zone is eclipsing hard payline. The gripe some of us are having is why this sudden change without giving sufficient explanation. Now, don’t give your boiler-plate answer that its NCI money and they can do whatever they want.

    Like

  9. EFGH Says:

    ABCD…HV did explain it here (http://videocast.nih.gov/summary.asp?Live=9846) see minute 28 or so. Maybe this isn’t to your satisfaction but it was explained.

    Now whether NCI should provide a public rationale for why it funded each grant outside the 7% payline is an interesting idea. Send in a FOIA request and see what you get.

    Like

  10. drugmonkey Says:

    The grey zone got increased under Varmus ( 8%tile points for established, 15 points for ESI) which is unprecedented. The gray zone is eclipsing hard payline.

    As I mentioned, I am pretty skeptical of any claim* that any IC did or does adhere strictly to the peer review order. It is not impossible that this is or has been so, but it is impossible to verify absent data like NIGMS publishes. In any case, there are for certain sure some of the ICs that have essentially always operated under the model that Varmus just made explicit for the NCI. The official line** has been that they do not have a payline–which is nonsense***. So to reiterate my point in this post, this type of approach should not be news to any NIH-funded investigator who has her head even modestly in the game. It may be a change of degree but it is not a change in kind considering the entire set of ICs.

    Now, the “eclipsing” claim? You can’t possibly know that. I refer you to the funding data published by the NIGMS and the long experience of those of us who have operated under funding from ICs that have this supposed no-payline approach in the past. All logic suggests that we must accept this as the likely outcome for NCI until and unless we see data or evidence that suggests otherwise. And these NIGMS graphs show quite strongly that the big signal continues to be the percentile ranks which emerge from initial peer review. The contribution of out-of-order pickups is likely to be much lesser. So your “eclipsing” description is very likely to be overblown.

    I am curious as to how you come to your conclusion based on your read of the available evidence.

    The gripe some of us are having is why this sudden change without giving sufficient explanation.

    “Sufficient” is of course in the eye of the beholder. But I thought the NCI official policy description was very clear. Much clearer in fact than policy from the types of alleged-non-payline ICs that I am referring to in these comments. What are you expecting here?

    *Even PO claims are suspect because we have seen a constant stream of data analyses from NIGMS and now OER in general that make it very clear (to this observer) that there is a ton of information that the ICs simply never review.

    **and your line of discussion regarding the current NCI approach shows, in living color, precisely why ICs are loathe to be upfront about their policies.

    ***I’ve had numerous conversations with POs which extract something along the line of “we’re going to fund everything to about X%ile this round so you are in okay shape” from these alleged no-payline ICs.

    Like

  11. kevin. Says:

    Still you got to feel bad for those few people at 6-8% who got dropped. I guess a reason could be those applications were to people who already had a larger grant and that the worry of redundant funding was present.

    Like

  12. Drugmonkey Says:

    Of course I sympathize. I just don’t think it is evidence that the system is corrupt, incompetent or broken.

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: