NIH pilots special scrutiny of PIs with $1.5 M in total costs.
May 18, 2012
Thoughts on NOT-OD-12-110:
The threshold of $1.5 million total costs. How’s that break down? Well if you are in a consensus ~50% overhead state university, let’s see…Thats FOUR full-modular awards. But let’s be clear, odds are you got cut by at least a module per award so that’s only $900K direct..you get to be in a University with about 70% overhead and you are still clear. What bout the much-rumored 100% overhead small institutions? well, you get three R01s before you go under strict scrutiny.
I do wonder if this will satisfy all the “kill the rich” voices? Will they see this as the NIH taking them seriously or as a meaningless sop?
Next question, this is just identifying special Council level review…No guarantee that any grant will ever be blocked because the PI has too much $$. No guarantee that negotiations wouldn’t be made either. “Say, PI Jones, would you please put some more junior colleague on as titular head to one of your other awards so we can give you this one”?
new MultiPI awards won’t trigger the scrutiny unless all PIs trigger the threshold. Hello courtesy “multi”PIship!!!
It may possibly change some people’s strategy so that they work harder to distribute effort around to other people’s awards in small percentages. Like junior PI is going to screw BigCheeze over on the agreed upon part of the direct costs? No worries there.
How are study sections going to respond to this. Will they take this as the NIH saying “This is our threshold for being worried about too much money. Now shut up about this for anything below this amount.”? Or will they take this as encouragement to think about lab size even more when they are reviewing the grant in front of them?
DrugMonkey 
May 18, 2012 at 11:55 am
The number of PI’s with 4+ R01’s (or >$1.5m in NIH funding) isn’t that large, right? If so, this smells more like a PR exercise more than a genuine effort to manage a reduced budget.
LikeLike
May 18, 2012 at 1:32 pm
Of course it’s pure PR. And pay careful attention to the trigger:
applications from Program Director(s)/Principal Investigator(s) [PD(s)/PI(s)] who already receive in excess of $1.5 million per year in total costs
This means the extra scrutiny isn’t triggered unless a PI *already* has > $1.5 million and is seeking *even more*.
The relevance of this to overall NIH funding boils down to the total dollar amount of competing awards submitted by PIs who already have > $1.5 million *and* which receive fundable percentile scores in peer review *and* which Council pulls out of the paylist. This is not going to move the needle on paylines or success rates *at all*.
NIGMS has had an even more stringent “extra scrutiny” cap for years, and if you look at their funding scatterplot from 2010, it looks like maybe *one* grant with a percentile below 10 got pulled from the paylist, and no one would ever (obviously) confirm that it was because of the “well-funded” scrutiny and not because they were proposing making human-chimp hybrids or some such shit.
The purpose of this is to keep the torches and pitchforks at bay, not to alter the fiscal structure of NIH funding patterns.
LikeLike
May 18, 2012 at 1:47 pm
Just left this comment at Rock Talk:
LikeLike
May 19, 2012 at 4:59 am
So… wait, I don’t get it. Rich hot-shots have an advantage of some sort?
LikeLike
May 19, 2012 at 5:41 am
Yes miko. Going by the internets, once you get to 3 grants they pretty much just hand over the 4th, 5th and 6th ones to anyone for the asking.
LikeLike
May 20, 2012 at 3:56 am
DM- I think you mean “eat the rich”.
CPP- solely from NIH perspective, extremely successful PIs getting a bridge from their institutions from time to time is basically ‘free money’ (albeit not a lot). That said, in conjunction with other trends (more soft-money assistant profs), it would basically represent an overall shift of risk to NIH- that is, institutions will pony up if and only if they know they’ll get more NIH dollars out of it it, while NIH has to support all the least tried-and-true researchers.
LikeLike
May 21, 2012 at 6:06 am
This is all just a load bullshite really. I mean if the NIH insists on this kind of micro-management, then they should at least make a real attempt at it. This is a bit like cutting the NIH budget to pay for the US government debt – it is a drop in the ocean and will make no difference at all to paylines. In fact paylines will probably continue to decline despite this PR exercise, especially in FY13 where nobody seems to know their asses from their elbows.
LikeLike
May 21, 2012 at 6:22 am
Yep. Do the math:
From Sally Rockey’s blogge post: “To give you an idea of the scope, for applications going to May councils that fall within the top 25th percentile, we identified approximately 70 applications that fit the criteria above.”
First, many of those 70 applications within the top 25th percentile aren’t going to be funded regardless, with paylines in the low teens. So let’s assume that half of them are competitive for funding: 35. And let’s assume that each of these is around the average R01 total costs (I think it’s around $400,000). That puts a total of $14,000,000 dollars into play.
Now, let’s ask the question how many of these applications are actually going to be recommended by the various councils to *not* be funded. Councils really don’t like to tinker with initial peer review, but let’s be generous and assume that they actually put the kibosh on half of these applications. So that puts $7,000,000 back into the pool for funding of competing awards. Now divide this back by $400,000 per grant, and you’ve got 17.5 extra competing grants added back to the funding pool. And that’s for the *entire* NIH, which has over *twenty* ICs.
Bottom Line: fewer than *one* additional competing R01 can be funded on average by each IC as a consequence of this policy.
LikeLike
May 21, 2012 at 7:13 am
So if this is the case, who on earth do they think they are fooling with this shell game?
LikeLike
May 21, 2012 at 7:26 am
Yeh exactly. What is even more depressing is that all this exercise really does is stir a (mostly distracting) debate about big labs vs. small labs and encourage this kind of “woe is me” rhetoric that is starting to fly around. As a young guy trying to get his first NIH grant, I understand the frustration, but the issue here is not about big labs taking all the money. Sure, there are big operations that have overlapping grants that they probably don’t need, but to focus on that is missing the point entirely. The fact that the NIH does not appear to have done the same math as PhysioProffe really makes me wonder whether they have any idea how to squeeze more out of a tighter budget.
I will get slaughtered for this, but I honestly am starting to think that the worst thing that could happen now is for congress to increase the NIH budget (over inflation) again in the next few years. The NIH has to continue to find ways to fund great research with the money they are given and without relying on year-on-year increases in funding. It just seems like it is the only sustainable way to go.
LikeLike
May 21, 2012 at 7:27 am
“So if this is the case, who on earth do they think they are fooling with this shell game?”
Themselves!!!!!!!!!!!!!!!!!!
LikeLike
May 21, 2012 at 7:33 am
The fact that the NIH does not appear to have done the same math as PhysioProffe really makes me wonder whether they have any idea how to squeeze more out of a tighter budget.
Of *course* all extramural staff at NIH have done the math and know this policy is not quantitatively relevant.
LikeLike
May 21, 2012 at 7:38 am
Hit send too soon: What they also know is that labs with substantially more than a single R01’s worth of funding are much more productive and creative than the single R01 labs. And they know that if they lower the threshold triggering “special scrutiny” low enough to make a quantitative difference to paylines, they are going to start hurting those highly productive/creative two and three R01 labs.
They know very well that to substantially shift the balance in favor of funding many more single-R01 labs is going to dramatically reduce the overall productivity of the NIH extramural program.
LikeLike
May 21, 2012 at 8:25 am
And they know that if they lower the threshold triggering “special scrutiny” low enough to make a quantitative difference to paylines, they are going to start hurting those highly productive/creative two and three R01 labs.
Well, that or they are going to have to make the calls between “highly productive” and “too-much-overlap” labs for themselves at the Program level. And this is 1) a fool’s errand to predict and 2) a massive pit of “bias” complaining.
LikeLike
May 21, 2012 at 8:52 am
Even if it made financial sense, which it doesn’t, all this chat about “special scrutiny” is just so vague it is hard to know what it actually means. If they want to stick to this approach, then they should at least have the balls to outright forbid applications from labs with x amount of R01’s, or x amount of dollars to their name.
If you think this isn’t implementable, then consider that the American Diabetes Association already forbid applications from PI’s with $500,000 or more in funding last year. Again, they forbid them, not put them under “special scrutiny” or anything silly. I don’t know how “effective” this has been, scientifically or economically, but at least they went all in on it.
LikeLike
May 21, 2012 at 9:10 am
“Of *course* all extramural staff at NIH have done the math and know this policy is not quantitatively relevant.”
The thing I find most disturbing about this is the idea that many meetings and discussions (presumably) by the NIH extramural staff have led to an outcome that everyone knows is not substantive, suggesting that the entire goal of the new policy is CYA. One has to wonder why they would even bother.
LikeLike
May 21, 2012 at 9:43 am
The NIH-wide policy to be piloted differs from the policy that has been used for years by NIGMS (http://www.nigms.nih.gov/Research/Application/NAGMSCouncilGuidelines.htm ) in several important ways. The NIGMS policy applies if FUNDING THE PENDING APPLICATION would put the PI over $750,000 in DIRECT COSTS PER YEAR considering ALL CURRENT SOURCES OF FUNDING (not just NIGMS or NIH). Thus, the threshold relates to the pending application, is independent of the F&A rate at the institution, and depends on other funding (from NSF, HHMI, etc.).
NIGMS has examined this policy over the years and has considered performing a formal evaluation. However, a major challenge to such analyses is that the eventual outcomes are not all-or-none. In many cases where exceptions are made, the application is funded but at a substantially reduced level or with an agreement that the PI will not attempt to renew an ongoing, but soon to end, grant.
In my 7+ years as NIGMS Director, my impression is that this policy affected the outcome for approximately 25 grants per year (out of approximately 900 new and competing grants funded per year) although this is just an estimate based on my recollections.
LikeLike
May 21, 2012 at 10:19 am
Yeah, your way sounded way more rational, JB…. 🙂
…but still. If the goal is to fund someone else’s grant, this special pinkie-swear and reduced amount stuff just raises further question about magnitude of effect. Maybe that is the better question- how many projects got *saved* b/c of these policies? 5? 10?
LikeLike
May 21, 2012 at 10:23 am
The reason they bother, BugDoc, is so that they have something to say when RandomCongressCritter comes asking b/c s/hejust visited the only research U in the District and got an earful about Coastal U Congressional Districts sucking up all the money.
LikeLike
May 21, 2012 at 10:36 am
Drugmonkey: Again, I do not have real numbers (and they would be hard to get even with access to all available data), but I would estimate that, over time, the NIGMS policy freed up funds for perhaps 15 additional grants per year. Furthermore, general awareness of overall funding levels certainly affected decision making in the “gray zone”. While these are not game changing figures, they are real effects, particularly for the investigators on the edge who do get funded.
LikeLike
May 21, 2012 at 10:37 am
“my impression is that this policy affected the outcome for approximately 25 grants per year”
Affected the outcome is indeed the appropriate phrase here. It is important to note that the grants in question were not rejected in full and so I doubt the money saved from a single incident was enough to cover an entire R01, or even two.
The comment about the PI agreeing not to submit for a renewal of another grant is interesting – how is that managed or enforced? What is to stop them from just editing it a little and submitting it as a NEW proposal instead? Sounds just as dicey as the approach under discussion here.
LikeLike
May 21, 2012 at 10:37 am
“b/c s/hejust visited the only research U in the District and got an earful about Coastal U Congressional Districts sucking up all the money.”
Ah – I wondered what that sucking sound was. Somewhat disappointed to know that CongressCritters are so easily satisfied, but given the recent post over at GoodMathBadMath on the lack of understanding of statistics of certain government officials, perhaps not surprising….
Since the better defined policy over at NIGMS has been successfully executed (as far as I can tell) for a number of years, I’m curious to know why NIH didn’t choose to pilot that policy for all institutes.
LikeLike
May 21, 2012 at 10:39 am
“general awareness of overall funding levels certainly affected decision making in the “gray zone””
That’s interesting, but isn’t there a need for some transparency here? This applies to those on both sides of the fence – those above and below the payline. It is all a bit wishy-washy if you ask me.
LikeLike
May 21, 2012 at 11:23 am
Dave: With regard to your first comment, I estimated that the policy led to funding an average of 15 additional grants per year as I noted in my response to drugmonkey.
The agreements about not submitting proposals either as competing renewals or as new applications are monitored by the program staff involved. The referral system makes it very likely that any proposal covering the same science would be assigned to the program director who had covered the grant previously. I know this was monitored and am aware of no examples of this having occurred during my time at NIGMS.
With regard to your second comment, I described the paylist meeting process on the NIGMS Feedback Loop blog (https://loop.nigms.nih.gov/index.php/2011/01/28/the-funding-decision-process/ ). The relevant section is:
For each application, the responsible program director presents the scientific topic as well as factors such as whether the applicant is an ESI or new investigator, how much other support the applicant has (particularly if the application represents the only support available to the investigator), whether the Council has given us specific advice on the application, whether the scientific area is perceived to be particularly exciting, and how much other research we already support in the general area of the application. The other members of the unit listen to these presentations, and the group then produces a prioritized list of applications.
Thus, a relatively well-funded investigator may be less likely to be funded compared to an investigator with little other support, everything else being equal. While these are qualitative considerations, the funding recommendations are made by a group of committed scientists, each of whom has “skin in the game” in terms of support for the areas of science and the associated investigators in their portfolio.
LikeLike
May 29, 2012 at 8:38 am
[…] factor that more concerns me is the drive at the NIH to kill the rich. We've been discussing this set of proposals that are targeted at making sure those who are successful at present don't […]
LikeLike
August 21, 2012 at 8:47 am
[…] that was fast. Quick on the heels of the pilot study conducted for the summer Council rounds, the NIH issued NOT-OD-12-140. In September 2012, NIH will […]
LikeLike