Thoughts on a proposal to escalate grant-acquisition difficulty
November 9, 2009
A recent post noted the decision by the NHLBI to adopt a payline policy that varied by grant revision status. The new R01 submissions would be subject to a 16% payline, the first revision to a 9% payline and any left-over grandfathered second revision A2 applications to a 7% payline.
In the course of discussion a reader proposed that what we really need is for the NIH to grade the payline based on how many grants a given PI already has. Commenter qaz said:
Maybe it would be enough to share the funding around better – make the first R01 easy to fund, the second harder, etc. If we made it possible for people to be funded at the 25% range (or even below that) if they didn’t have any other grants, then maybe it wouldn’t be a problem.
This idea was seconded by Principle Investigator.
Knowing a landmined topic when I see it, I had a few observations.
First off we need to point out that some people’s job category is soft-money and all research. Meaning that, IMNSHO, one $250K / year full-modular R01 is not anywhere close to enough. Two is about the starting bid. I already hear the response from the hard-money crowd- nyah, nyah it’s their own fault for taking that job category. Sure. But so what? Perhaps the NIH mission is best served by those who focus exclusively on research and don’t have the headaches that come along with hard money (read: professin’ responsibility) jobs. Perhaps the NIH (and therefore the taxpayer) get more out of soft-money PI’s second R01 than they do out of the first R01 of hard-money Prof. So perhaps we should make the hurdle harder for the hard-money person.
Second, as we can now review for ourselves, one R01 award might be for $250K / yr /5 yrs…but another might be for $498K / yr / 5 yrs. And another might be for $175K for three years. I doubt it is much of a stretch to point out that the older and more established labs are more likely to have the bigger awards that are over the modular limit. This is certainly what I observe in grant review. Somehow I doubt that the commentariat is suggesting that n00b PI with a $175/yr/3yr starter R01 should face the same hurdle for grant #2 as they want Prof Bluehair with the $487K /yr/5yr in year 20 to face when applying for another R01.
Third, following on from that last comment, how are you going to deal with the timeline and practicalities of the grant cycle? It would not be uncommon for a very well-established lab to have a core R01 that has been continued repeatedly. Sometimes in year 15, 20… or 25. They might then have a random 3 or 5 yr R01 now and again. Suppose they happen to hold one of these more peripheral awards when the renewal comes up…should the hurdle be higher? Will this make some silly shenanigans happen like turning back an award’s outyear 4 or 5? I could see a situation where you’d have one year left on the random award and a score for the (likely larger) renewal that sat between thresholds. Too high for the 2nd award but right in there for the 1st award (again, under the proposed sliding scale).
Fourth, what about alternate sources of funding? Should the investigator loaded up on DoD and NSF grants, or with HHMI moneybags, be treated the same for the first R01? the second? …the, um, fifth? (Yeah, I know some HHMI dudes in this zone…)
Okay, that should be enough to think about*. May we have a defense of your proposal, qaz and Principle Investigator?
__
*although I feel certain my readers will come up with some new fun stuff to fight over
DrugMonkey 
November 9, 2009 at 7:03 pm
The “problem” as identified by QAZ and PI is apparently overconcentration of resources. DM, do you disagree that this is a problem, or do you agree that this is problem and disagree with the proposed solution?
LikeLike
November 9, 2009 at 9:18 pm
My understanding is that the statutory authority for NIH to award grants requires that they be awarded on the basis of “scientific merit”. While things like New Investigator and resubmission status payline differences can be justified as relating to “scientific merit” because they arguably redress bias in the study section merit assessment process, and administrative agencies are entitled to tremendous legal deference in their interpretation of the statutes that govern their activities, it is difficult to see how differential paylines depending mechanically and non-discretionarily on the existing funding of the PI are even arguably related to “scientific merit”.
LikeLike
November 9, 2009 at 9:22 pm
On one had to the victor go the spoils as is the American way, but we are not talking about a level playing field. There is really no way to equitably administrate this idea of making subsequent R01’s harder to get.
LikeLike
November 9, 2009 at 9:27 pm
FYI, some ICs–at a minimum NIGMS and NIMH–have explicitly published funding strategies of discouraging award of additional grants to “well-funded” PIs. However, they also explicitly allow for “scientific merit” to trump this policy and do not apply it mechanically.
LikeLike
November 9, 2009 at 10:04 pm
I’ll acknowledge that there are a small number of investigators who have demonstrated so much productivity and discovery over many years that they do, in fact, deserve to get more NIH support than most other investigators. If you don’t admit this, you are a communist (just kidding, sorta).
The problem arises when investigators with 3 or more RO1s or equivalents are not actually producing 3-4 more times the research. Rather, they are “cross-posting” their discoveries from one grant to support another. This creates the illusion that they are incredibly productive, when in fact what they are doing is skimming the cream of the best research from all of their grants. In this case, the concept of meritocracy has been undermined.
In an earlier DM post qaz identifies a key consideration: biodiversity. Is it better to fund yet another study from big-ass lab, or is it a better investment to fund smaller operations working in more isolated ecosystems?
My bottom line: there are diminishing returns on RO1s above 2-3 (depending on the hard/soft money issue identified by DM). It is perfectly fine to allow productive investigators to hold 3 or more RO1s–but it is also OK to raise the bar for them. If a 4th RO1 is scoring in the top 5%, then I’m willing to consider that perhaps the study section was impressed by the science, not the pedigree.
LikeLike
November 9, 2009 at 10:14 pm
One way to address this issue is to require a reasonable percentage of effort for the PI on a grand. This would allow folks with fewer non-grant related responsibilities to have more grants, and expend greater effort on each. For example, if we set PI effort at 30%/RO1, it would cap # of grants to 3 (well, and given the requirement that you cannot work to obtain federal funds while being paid on federal funds, it would really cap it lower than that). This configuration would also take into account outside resource (because non-NIH resources would also require effort).
And, apologies to Drugmonkey, but I think the only PIs who are supported 100% by NIH should be NIH employees. A more rigorous enforcement of existing federal law (i.e. can’t write grants on the federal dime) would be a good way to squeeze the system in the right direction. PIs could ask for more money within a single grant, but then, they’d be forced to justify the higher cost with higher productive within a single application.
LikeLike
November 9, 2009 at 10:44 pm
OK. Let’s take this apart.
First, the issue (as has been mentioned here before) is that the loss from 1 to 0 grants is much more devastating than the loss from 4 to 3. More importantly, it is much easier (in the current system) to go from 3 to 4 grants than it is to go from 0 to 1 or from 1 to 2. There are lots of reasons for this. I am going to assume that DM agrees with me on those statements. (If not, I’ll defend them later.)
Second, concentrating money into a few labs produces calcification and starves out diversity. As with most ecosystems, diversity actually produces a healthier economy/ecosystem with more innovation, more niches, more stability, and more productivity. Note, this does not mean that one can’t do more work on two R01s than on one. In fact, I can say from my own experience that a second R01 increases productivity by more than 100% of the first. But there are also diminishing returns. How much does it help the scientific enterprise to give an HHMI lab a third R01? Is that worth starving a small lab over?
Third, the problem is that we have a “death wall” where if you run out of money, you die. (Or your lab dies, which is not the same, I guess.) This means that when a lab reaches $0, it disappears. Extinction is forever.
So now we have reasons for encouraging diversity in labs and for preventing labs from dying.
Now on to the proposal. There are four issues raised by DM. (1) That some people are soft money and one R01 is not enough. (2) R01s are different sizes. (3) We really need two R01s to be a continuous funding cycle. and (4) Counting R01s is not a valid way to count current lab health.
I’m going to ignore issue 1 because as neurolover (#6) points out, those people should be NIH employees. I don’t support the soft-money system and I won’t defend it.
Issues 2 and 4 come down to the issue that we shouldn’t measure money in R01s. We should measure it in $. That’s fine. Makes sense to me.
Issue 3 raises the question of whether this five-year grant cycle is really the right thing anyway. (I don’t think it is. I’ve said before and I’ll say it again, the grant system is a terrible way to fund a lab and do science. We waste tremendous time writing grants, waste time reviewing them, and even worse, waste time rewriting grants. We need a system based on a person’s record and productivity, in which its easy to keep going so we can take the time to do the science. I’ve made this proposal elsewhere so I won’t repeat it here.) But as I know that’s never going to happen, it would not be hard to construct a system where you have one primary grant, renewing it has the low threshold, and then the secondary R01s have the high threshold. My point is that this is not a difficult issue to fix if we do it right.
I should also point out that my original proposal was very in line with Beaker’s comments (#5). It is perfectly possible to have 4 R01s, but it should get harder each time.
As has been said by many people before, the concept that there is an identifiable “scientific merit” distinction between anything in the top 25% of the submitted grants is ludicrous to anyone who has ever experienced the joy that is study section. Therefore, I do not think it would be difficult to create two categories “scientifically meritorious” (i.e. fundable) and “scientifically not meritorious” (i.e. not fundable). We could even have a category of “if we can” in the middle if you like. But I don’t think that would endanger NIH’s mandate of funding based on scientific merit.
LikeLike
November 9, 2009 at 10:53 pm
[I]t is much easier (in the current system) to go from 3 to 4 grants than it is to go from 0 to 1 or from 1 to 2.
This is totally false.
LikeLike
November 10, 2009 at 12:38 am
OK. So we are scientists here. Does anybody have any data on all the crap that we are discussing? I know that it would be more “fair” to give to the poor and take from the filthy rich, but is there any clear evidence that the Glamour Labz are actually less productive per $$$ than the small labs? Do we actually need to increase the supposed diversity by killing the “big lab”, or are the “big labs” the ones where the most innovative science is actually being made? Last time I checked there were some really cool pubs coming out of the “big labs”. Do we need to explicitly bias against them or is the process of natural selection creating the right “big lab”/”small lab” equilibrium? We are entering into a discussion of policy without ANY evidence of what would be most beneficial for science as a whole. If anyone has any numbers, please quote them.
LikeLike
November 10, 2009 at 4:57 am
Dude, what are you, some kind of far-right-wing Randian plutocrat class-traitor freakazoid? VIVA LA REVOLUCION!
LikeLike
November 10, 2009 at 7:07 am
“We are entering into a discussion of policy without ANY evidence of what would be most beneficial for science as a whole”.
The evidence is in the CRISP/Reporter. Unfortunately, the numbers cannot be summed up because CRISP (for unknown reasons) interrupted publication of amount of dollars/year to any given PI from around 1997 until 2006. The fact that some big labs have been getting between 15-20 million dollars/ year, while small ones( in the same field) have been getting $300,000-500,000, or have been shut down is an indicator that we need to look at the numbers and at the “really cool pubs coming out of the big labs” and, therefore, at the policies.
Who knows if those “really cool pubs came out of big labs” at the expense of premeditated killing of some innovative science going on in small ones?
LikeLike
November 10, 2009 at 11:19 am
Let me get this straight. No soft money job sector, thus all NIH applicants would have to be on hard money support from their institutions. Teaching profs and the, what, 10% or less that could be supported on endowed chairs.
Plus their grants would be limited meaning fewer postdocs and smaller labs.
Postdocs who would have no where to go because there would be no jobs available* under Section I of the qaz code.
Our doctoral level scientific work force would shrink by a LOT.
I don’t see how this is good for the NIH mission.
*Sorry but I don’t see where making the chances of each new hire getting a single R01 slightly better (or even a slam dunk) is going to encourage a lot of new hard-money job creation at Universities.
LikeLike
November 10, 2009 at 11:23 am
There is perception bias when you look at your favorite middle-of-road PI with 4 grants and don’t see where she is any different / more productive than you are getting beat up for the second award. You don’t see all the 2-3 grant folks that could never get a 4th. Track back and the same argument- you don’t see all the people who tried like hell and could never get that second R01 going.
Longwinded way of saying I agree with PP to the extent that we don’t really have the data to make assertions about which 1-grant step is hardest.
LikeLike
November 10, 2009 at 11:24 am
Yeah! “Who knows…?” I, for one, don’t. The argument that some labs are getting $20M doesn’t mean shit! Maybe they are doing clinical studies of a new drug for goodness’ sake. Gimme stats, please. Not that I wouldn’t like to see small labs funded because I think that the overall “ambiance” is much better in a smaller lab than a big-ass giant, and my intuition is that creativity also fares better in a more intimate environment, but that doesn’t mean that objectively the giants are not giving NIH a bang for their buck.
LikeLike
November 10, 2009 at 11:34 am
niewiap:
You have to define some terms. What is “bang for buck”? in your view?
-GlamourPubs?
-more pubs?
-more citations?
-obvious public health relevance?
big labs seem to be necessary for sustained glamour publication but comes at the cost of fewer pubs and, more importantly in my view, much of the underlying science never seeing the light of day. underlying science that someone else then has to go on and repeat in many cases.
more citations goes with Glamour and goes with field size and cite practices of course but may be meaningless in the long scheme of public health significance. I can point to some pretty pedestrian pubs (in terms of the journal at the time) that are the absolute *cornerstone* of modern treatments and/or the absolute cornerstone of a whole host of following research. sometimes these papers are a one-off for various reasons. small lab doing very little work or a random grad student project in a big lab or whatever.
LikeLike
November 10, 2009 at 11:58 am
@DM:
I am not defending the big lab as such. I just think that the big lab is not necessarily the evil it is portrayed to be. Most of what I have seen and what you are saying is anecdotal evidence, and I’d like to see some real numbers. The argument that some of the most fundamental papers in biology were published in small labs is a strawman – probably even more fundamental papers were published in big labs. I’d like to at least see some very dirty stats, say, NIH total funding in $ vs. number of publications in at least half decent journals (say, above IF 5). Yeah, yeah, I know IF is shit, but that’s beyond the question. Maybe NIH funding $$$ vs. time-shifted number of citations of non-review papers from the lab. Show me any graph that supports the small lab model and I’ll be satisfied.
On a different note – a big lab may actually be more likely to follow more risky paths – they have so many projects that they can afford to have one fail. A small lab, on the other hand, will be more conservative with their choice of topics – they can’t afford a completely out there project to fail on them.
LikeLike
November 10, 2009 at 12:35 pm
I think (?) I am acknowledging that you will be hard pressed to find “the graph” …in support of big labs OR small labs as being the OneTrueWay.
I suspect we all acknowledge that exceptionally well funded labs under the direction (or at least thumb) of one person can be both tremendous wastes of money and the only way some really big kewl science could possibly have been accomplished.
All we are carping over is relative ratio, yes?
But there is also the issue of catch-22 and opportunity cost and yes, for that we are left with anecdote and analysis of human behavior. Opportunity cost I dealt with in the OP.
On the catch-22 front, remember that those who are proposing to revamp the entire system are proposing that the small-potatoes-PIs will be in a changed environment with more long-term stability of funding. People such as myself might be advocating a change in the scooping/first-to-publish thing that slants toward the huge labs. I just heard from a reader this very week about yet another story of big lab vs small lab in the GlamourWarz. In this case the smaller PI managed to sneak in a GlamourBlip ahead of the ScienceMonstrosity. It is no stretch to observe that sometimes if the smaller PI who had the idea and some of the data first gets beat to GlamourPub, that’s a death knell for the career (at that level).
LikeLike
November 10, 2009 at 1:00 pm
Great proposal, qaz, but it’ll never happen, because the ones with the most already would lose the most. That’s not the way things change at NIH. Look at the way most of ARRA money was just handed out. Did the NIH use the opportunity to spread the wealth? No; for the most part the rich just got richer.
Similarly, the lowered payline for revisions increases funding rates for those who come in right away with the strongest grants — e.g. the most established and already successful investigators.
The only way NIH has a chance in hell of changing in regard to the point under discussion is when someone actually measures productivity vs grant money, gets Nature or Science to publish the inverted-U curve that will undoubtedly result, and someone shows it to congress and explains that multiple awards and big awards squander tax payer money, and demands action. NIH policy will have to change or they will face a skeptical congress when it comes to budget increases.
LikeLike
November 10, 2009 at 1:23 pm
“well, and given the requirement that you cannot work to obtain federal funds while being paid on federal funds, it would really cap it lower than that”
Who knew? I always wondered why we had a grant development code to enter on our time sheets at appropriate times.
Is this enforced? 😛
LikeLike
November 10, 2009 at 1:27 pm
@DM:
Yes, we are carping over relative ratio, but we will not come to any conclusion without data.
@A reader:
Amen!
LikeLike
November 10, 2009 at 1:49 pm
Is this enforced? 😛
yes and no. some places have been audited and forced to always maintain some 5% or whatnot off the NIH dime. Some places have a voluntary version and (perhaps like yours?) a way to pay for that time. Other places have a voluntary version but no way to actually pay for the time off federal grants. in the latter case, the institution is protecting itself (I guess) by getting the PIs who work there to lie to them. I wonder if the audit and forced 5% cases started out with this lying system…
LikeLike
November 10, 2009 at 2:44 pm
There was a Qui Tam (i.e. “whistleblower”) suit about faculty effort against Cornell (in 2006) that ’caused official revisions of how effort was defined/reported at universities that got worried about compliance issues. Cornell settled (for about 5 million) and so did Northwestern and a few other universities. One issue was the absolutely clear rule that one cannot write grants while being paid by federal funds (it’s an offshoot of the idea that you’re not allowed to “lobby” for federal funds while being paid by federal funds). Of course, many “grant” activities can be reclassified (i.e. a figure prepared for both a grant and a paper can be considered as having been prepared for the paper, which is a allowable activity under federal funding).
At our university, an effect of the settlements has been making faculty take little math quizzes on faculty effort that go something like this: You work 60 hours a week. 50% of your effort is paid off grant A. How many hours do you work on Grant A projects? and a mad rush towards finding the 5% funding (that compliance officers have determined will keep you clean)to write federal grants. So, I guess if you work 3000 hours/year (50 weeks, 60 hours/week), you’re allowed to spend 150 hours writing your grant(s).
LikeLike
November 10, 2009 at 3:02 pm
The grant writing time is a rake off the F&A costs. In non-profits they can put a slush fund for “development” etc. into the cost base.
LikeLike
November 13, 2009 at 1:10 am
in my field most of the important breakthroughs are done by the big labs. They have the expensive equipment to do things that the small-potatoes-PI could never afford to buy or pay for. they also did take on riskier projects because one project failing was not going to be the end of the lab. Meanwhile the small-potatoes PIs are trying to play it safe being motivated primarily by getting their next grant so they can keep their heads above water.
I see the small-potatoes PIs (in my field at least) as people who are more about ego than about doing the best science that they can do. they want to run their own ship and be the boss. fine. But realize that some times you can accomplish so much more as a scientist if you have access to top-notch (i.e. expensive, usually) facilities, money to throw around at new ideas, people at your level (not trainees) as your primary colleagues, and the benefit of professional technical support staff, rather than relying on a small budget and a handful of grad students and your collaborator’s grad students to do everything.
Under the current system the rich get richer. But so what, the poor perceive this as a problem only because they want to get rich too. Why don’t the poor instead quit competing with the rich and go work for them? You might even do better science that way.
LikeLike
November 13, 2009 at 2:05 am
What is your field, Anonymous?
LikeLike
November 15, 2009 at 5:03 am
@ A Reader
My field is a new one, with ancient roots. It is very interdisciplinary. It is called: Making-a-difference-logy. Its impetus came from a recent discipline called Bushology that pushed me strong into publishing by means of ballshitting. It also helped me cutting-edging by being all Committeeing-Awarding friendly with friends of friends.
See, it’s so elite of the elite that only few people like us have the brain and business foundation to understand it. But we are making a difference trying to enlarge our world of intellectual influence. We, the first of the firsts, don’t kill each other but collaborate. That’s key, you know. But we do fight in labcoats to demonstrate that we know our science.
LikeLike
November 15, 2009 at 9:31 am
Gee, Anonymous, I didn’t expect such a helpful answer to my honest curiosity. Thanks!
LikeLike