…and if I have it right, the Director of a given IC is really the one who makes the decision to fund or not fund a given grant proposal…and even the input of their own Program Staff is only advisory.
Some comment over at writedit’s thread on paylines is incensed about the NCI:
how are the NCI officials better qualified then the peer reviewers to judge the importance of these “selected” studies?
Because They. Have. A. Different. Perspective. on the fields of science. A broader perspective. In the best of all worlds, the Program Staff have the opportunity to step back and look at the larger trends within the science that is within their jurisdiction. To see where current fads have left holes in their portfolio. To perhaps take a risk where the peers are conservative. To identify the duplications and overlaps within their own portfolio and adjust accordingly. To worry about the next generation of scientists. Etc.
How can individual scientists, or even a panel of 25-30 of them, possibly take the long view? They cannot. So there is a role for Program. We can debate whether a funding agency should be sensitive to the long view, balance of effort, inherent self-referential conservatism that emerges in science now and again, etc. I’ll come down squarely in favor of breadth on that one. But let us not pretend they have no functional role.
They should not be make funding decisions. They are messing up the once well established system.
Yes, yes they should be making funding decisions. This is the job of Program, actually. They are part and parcel of the “well established system”.
I’ve touched on this ever so briefly before, see Program Interferes with…. and NIH Administrators Ignore…
Oh glory…since I started writing this post, the commenter doubled down:
If the grants do not fit the NCI portofolio they should not be sent forward for peer review in the first instance; and more importantly these same grants that do not fit the portofolio in the first place should not be sent back for an A1 revision which consumes a whole lot of funds to generate additional preliminary data. I think this suggests that not all the decisions are made by well qualified individuals at NCI.
and an echo from yet another commenter….
If NCI had told me that my app. did not fit their portofolio, I wouldn’t have wasted freaking 4 yrs submitting A0 and A1( plus wasting peer review efforts). And before submitting my A0, I had even consulted my SRO. I’m pissed that NCI changed rules in the middle of the game
HAHAHAHAAHA!!!! This commenter no doubt would be raising a big stink if his/her grant got rejected without even going out to peer review. But the underlying principle that Program should be highly pro-active about refusing grants before they even get reviewed is stupid. There is no chance for reviewers who are more expert in the science to point out whether it fits the mission or not. Program staff are not omniscient. They need their extramural scientists to educate them. Not to dictate their job priorities, not at all. To educate. To provide a portion of the knowledge, information and evidence that Program staff require to do their part of the job. This is at the root of the investigator-initiated science funding system is it not? It is our job to make our case. The job of peer review to provide one viewpoint on that case. The job of Program Staff to provide another set of inputs.
I like this. I like the ability to make my case for what I am interested in studying. I would be far less enthralled to have to always fit into some pre-existing set of Programmatic interests. I think our friends over at writedit’s blog would be similarly distressed if there were a more heavy handed triage of applications prior to review.
The only way that would work is if someone at the Program level does a lot of triaging on the basis of the Abstract. Because I guarantee you the POs are not going to be reading all the apps in detail under such a scenario. Too many applications and too few POs. And if there were enough of them to do so? We’d all be crying foul about why it took 10X as many Program staff as they have now! Think of how many grants that would suck up. And they still couldn’t offer the kind of specific expertise that the current peer-review system can muster.
and back to the original commenter…..
I would scream bloody murder if the grant was peer reviewed, fell in the upper 10th percentile and then was told that it did not fit the portofolio…….to an extent that is what is happening at the NCI….but you are entitled to your opinion
Of course the grey zone has long been familiar to those of us who seek funding from some of the other institutes. I have my doubts, of course, about assertions that some of the ICs have or had a policy of sticking strictly to the outcome of the initial peer review. But….perhaps this has indeed been the case at NCI. And the Chicken Littles can be excused, a trifle, for not knowing how it goes down. We have only the hard data from NIGMS but these graphs fit very well with my subjective experiences as an applicant, a friend and colleague of applicants, watching what emerges from a study section on which I served, talking with POs, etc. Take this FY2010 R01 funding outcome graph from NIGMS as an example:
Open rectangles depict the number of grants reviewed, the dark bars the number funded. The X axis depicts percentile scores that emerged from initial peer review. What this shows is that genuine “skips” are relatively rare. There is a clear payline (formally published or not, you can see where it lies) below which almost everything gets funded. Above that line is the grey area. A zone above the payline (I note for those who are thinking that they “deserve” to get funded) in which only a subset of the grants get funded. Notice the trendline though? Even this is influenced strongly by the priority score/percentile rank, right? The chances of an application being funded as an exception to the payline increases as the score moves closer to that payline.
The NCI has apparently been talking a payline in the 7-8%ile zone and mollifying investigators that they will “consider” everything up to some 25 %ile for exception funding. Pickups. So the scenario raised by the commenter is asking about apps in the 8-10%ile range and their chances of being skippped over. Unless NCI takes a very different approach to exception funding, the chances for such a score are probably still quite good. Once you get a couple of bins away from the payline in the NIGMS data above (and check the sidebar at the NIGMS page for prior Fiscal Year trends) then the chances for any application funding get pretty slim.
It is difficult for me to understand how anyone can look at the distribution of funded/unfunded grant application in this grey area and think that they “deserve” to be funded with scores that are above the payline. I say rather that people should feel lucky to get the nod, given that few of the apps are being funded.
DrugMonkey 