NIAID publishes their FY2013 paylines
April 19, 2013
NIAID is one of the NIH ICs that actually publishes a payline. According to their website, as of April 19 the R01s from experienced investigators will have a payline of 8 percentile. The payline for new investigators will be 12 percentile. By way of comparison these were 10%ile and 14%ile in the prior two Fiscal Years for NIAID.
Mechanisms such as the R03, R21 and R15 will have to get a 20 overall impact score, or better, to fund but these are still listed as “interim” criteria.
So from a statistical basis, you need to have put in 13 proposals to NIAID this year in order to have a fighting chance to get one.
Lovely.
DrugMonkey 
April 19, 2013 at 2:25 pm
NHLBI is 6% for RO1s
LikeLike
April 19, 2013 at 2:38 pm
Is NIGMS still publicizing theirs or did that end after Berg?
LikeLike
April 19, 2013 at 3:26 pm
Such wonderful news!
The Budget Control Act of 2011 and Sequestration should insure such paylines for the next 5 years. But we still have a ways to go; after all, these paylines are still several points above zero.
And no one is coming to your rescue. Your darling in Congress, Arlen Specter, is all gone. Add to that incoming budget pressures from underfunded entitlements, and you are about to get hit with the Perfect Storm. My advice: Take action now and run for the hills, or you will finish end up like this guy:
http://discovermagazine.com/2011/apr/30-how-bad-luck-networking-cost-prasher-nobel
LikeLike
April 19, 2013 at 5:32 pm
These paylines look good compared to some ICs.
LikeLike
April 20, 2013 at 6:17 am
This is from the NINDS February Council minutes:
NINDS paid almost all grants from October 2012 Council from its FY12 budget allocation and is currently expediting grants to the 12th percentile. Dr. Landis indicated that the Institute hopes to be able to award grants to the 15th percentile; however, the ability to do so is contingent upon the outcome of ongoing federal budget negotiations.
I have been told by program staff that since sequestration was not reversed, the payline for FY13 will remain at 12%ile.
LikeLike
April 20, 2013 at 10:02 am
You are doing the math wrong again. Each proposal you put in has a probability of (1-0.08) of not being funded. If you put in N proposals the probability that they all won’t be funded is (1-0.08)^N which for N=13 is 34%, e.g. you have a 2 in 3 chance of winning.
LikeLike
April 20, 2013 at 11:23 am
And I would like to see the data of probability of funding as a function of number of grants submitted, which probably doesn’t follow the random probability rules. The research institutes lobby had a number like that. Does anyone have a quick and dirty way to get such an estimate? would a DM survey give us anything? How about internal surveys?
I’m guessing stopping rules might influence the stats, though maybe they shouldn’t (I.e. your rate of submission should not depend on whether you get a grant.)
LikeLike
April 20, 2013 at 11:39 am
Good article on Prasher. But, I think one that shows the skills needed to do science that get underplayed to the young and naive, rather than a systemic failure. Scientists do have to have an entrepreneurial spirit, sell their science, find collaborators and “investors” and ask for help and support. And, the have to have a fighting spirit.
Does science loose when people who don’t have all those characteristics in addition to brilliant analytic minds, great lab skills, and perseverance? Maybe, a bit, but many people have both.
LikeLike
April 20, 2013 at 12:02 pm
And I would like to see the data of probability of funding as a function of number of grants submitted, which probably doesn’t follow the random probability rules
This is where I have a problem with the “you need to submit 13 applications to stand a chance” stuff. It just doesn’t work like that because this assumes that each application has an equal chance of success, which cannot be the case when you are submitting 13 R01s in one year, and it doesn’t take into account A0/A1s, the investigator etc.
LikeLike
April 20, 2013 at 12:38 pm
Also, there is likely to be a decent correlation between the scores of a given PI’s grants. Not really high, but it is there. So the PIs who are accomplished and doing good science and good grant-writers don’t need to submit as many grants as the ones who need a lightning strike to get funded.
LikeLike
April 20, 2013 at 4:10 pm
Everyone who has a good run thinks they deserve it for their awesome science and excellent grantsmithing, PP. Until they hit a dry spell and then it is all about the incompetent and biased reviewers who make errors of fact.
LikeLike
April 20, 2013 at 4:14 pm
The NINDS thing sounds dodgy. Using end of the year funds to get a jump on the next FY? And sooooo not the thing to do to stop the grant churning.
LikeLike
April 21, 2013 at 6:06 am
Eli: of course you do realize that, by formalizing “fighting chance” into “1 in 3 probability of failing even though you wrote 13 goddamn applications”, you’re not exactly lifting our spirits.
LikeLike
April 21, 2013 at 6:22 am
The NINDS thing sounds dodgy. Using end of the year funds to get a jump on the next FY? And sooooo not the thing to do to stop the grant churning.
I disagree completely. The goal was to smooth out the effect of sequestration so that paylines did not have to plummet in 2013. This makes sense from an equity standpoint, as the vagaries of exactly when an application is submitted should not have a massive effect on likelihood of funding.
LikeLike
April 21, 2013 at 6:38 am
Also, there is likely to be a decent correlation between the scores of a given PI’s grants.
There better be. Otherwise, peer review is just a waste of time, and they might as well just pick randomly from the submitted proposals.
DM is correct statistically — each PI needs to submit on average 13 proposals. But DM is wrong from an individual PI perspective. As many have pointed out already, some PIs will submit only one or two and get funded, and others will submit several dozen before they get funded (or never get funded). The best predictor of future success is always past success.
LikeLike
April 21, 2013 at 11:16 am
@Eli: You are doing the math wrong again.
DM is doing the math right. So are you. You are asking different questions.
This scenario seems to be a straightforward description of a geometric distribution. The expected value for the number of applications needed to get the first success will be 1/p, where p is the success rate. In this specific example, 1/0.08 is about 13. Of course, as pointed out by TOD, if your success rate is better than most, then go ahead and substitute your own personal p in there. Good luck quantifying just how special you are, though. It might not be so unreasonable to expect that your probability of success with an individual grant is about the same as everyone else’s. Especially since this includes the decision making properties of butthead reviewers (there are always some in the study section, right?).
A bad thing about this math is that the curve is asymptotic – as success rates get lower, the number of applications needed shoots to infinity (n = 1/p).
Eli, you are answering a different question, which is really about the cumulative distribution instead the expected value – what’s the chance of having at least one success in 13 applications? I agree with your calculation. But, like others, I am not feeling oh so reassured by it. There is an implied value judgement here – at what point should I feel good about my chances? 2/3 after 13 tries does not make me happy. Let’s say I need to be 90% certain I will get a grant, in order to plan my career. Then I need 27 or 28 applications at a success rate of 8%.
If the success rate were 20% (and I’m doing the math right…might not be the case), this would fall to just 10 or 11 applications to have a 90% chance of at least one getting funded.
LikeLike
April 21, 2013 at 1:59 pm
This is what I am thinking…
1) Whatever the success rates, no one ain’t never gonna get squat unless they apply.
2) If NIH and NSF don’t want to deal with so many applications, they need to stop making it necessary to send in so many applications.
3) Statistically, I have about the same odds of getting an R01 as getting a paper into Cell, Nature, or Science. Which is better?
4) About 90% of the time people spend working on proposals could be spent watching TV or masturbating, with no overall loss of productivity.
5) If my time is worth $50/hour, and I spend 50 hours working on a proposal, then that’s worth $2500. If I buy $2500 worth of Powerball tickets, I have about a one in one-hundred thousand chance of winning. That’s still worse than NIH. But it gets closer the longer it takes me to do a proposal or the more my time is worth. So I should probably work at getting an NIH grant. But once I get that R01, I can invest all $250K of my direct costs in the Powerball. That will give me about a one in one thousand chance of never needing to apply for funding again. Not really high enough odds to make it worthwhile. Therefore I should apply for a non-modular grant so I can invest more in the Powerball.
6) Instead of writing proposals, maybe I should be day-trading or selling real estate and funding my science with the proceeds. Might be more efficient.
7) I should hire 13 postdocs and make them each write an R01. But I can’t pay them all off the one R01, so they will have to work for free. How can I get them to write proposals for free? Hmmmmm… I know: I’ll let them join the lab contingent upon ‘their’ (my) grant getting funded. And then while they’re in the lab I’ll make them write more proposals. It’s good training, right?
…I don’t think #7 is that original.
LikeLike
April 21, 2013 at 5:31 pm
Wow, that was the best post ever. Well done TOD. I’m printing this thing out and putting it on the wall of my office next to teaching and research awards that don’t matter ’cause we live in a “what have you done for me lately?” world.
LikeLike
April 22, 2013 at 5:17 pm
So from a statistical basis, you need to have put in 13 proposals to NIAID this year in order to have a fighting chance to get one.
Umm…. no. Your 13 applications are not independent or random events. If they are, then you shouldn’t be in science. The chances of making the payline are probably bi-phasic. If you’re not funded in your 1st application, your chances on the 2nd may be better because it’s amended*. The 3rd may be better too, because you’ve had the benefit of 2 reviews to focus on the most important parts. After a number of failures, though, the study section(s) doesn’t think your idea is significant and/or well executed, so your chances start going down. Now, given that you can’t submit 13 sequential applications in 3 cycles, there will be overlap and different study sections and all sorts of confounders to figuring out the probability.
* let A=success of 01, B=success of 01A1
You could say P(B)>P(A), but if the 01 is successful, then there is no 01A1, so you’d have to say that P(B|Ā)>P(A), i.e. the probability of a successful 01A1 given the failure of the 01 is greater than the probability of a successful 01.
Boom! Science, bitchezzzzzzz!
LikeLike
April 22, 2013 at 5:52 pm
I like option 4
LikeLike
April 23, 2013 at 1:17 pm
I’m a NI/ESI and missed the NIAID R01 by 1%tile. I got a 13%tile with a 20 Impact/Priority score. Vagaries of life, indeed.
LikeLike
April 23, 2013 at 4:11 pm
Ugh. Very sorry to hear that. Cross fingers for end of year pickup!
LikeLike
April 23, 2013 at 5:16 pm
Ask your PO to R56 you.
LikeLike
April 23, 2013 at 6:35 pm
Good point Whimpie!
LikeLike
April 23, 2013 at 7:16 pm
What’s the cost benefit of 2-4 weeks total angst to write each of those 13 grants versus the miniscule effort to buy powerball tickets. Whatever to the notion that grant writing helps you focus your research plan, when it has become more salesmamship and going with the grain than proposing true innovation. At least powerballs gives you tiny sprit of endorphins before drawing. Endless grant writing is soul sapping and still a pricey ticket to play the NOA lottery……
LikeLike
April 24, 2013 at 3:57 am
A big billboard on my way to work advertises that the current powerball jackpot is $104 million. Isn’t that the amount that Obama promised for the brain mapping project? I momentarily entertained a fantasy of winning the jackpot, endowing my lab with the entire proceeds, and telling the media that Obama’s proposal is mere peanuts.
LikeLike
April 24, 2013 at 5:49 am
I momentarily entertained a fantasy of winning the jackpot, endowing my lab with the entire proceeds
Minus the IDCs……of course
LikeLike
April 24, 2013 at 6:53 am
I already play the powerball with the hope of creating an endowed position for myself. My PI has admitted occasionally doing the same thing. What I’m saying is you people are behind the times.
Between taxes and IDC’s, funding an entire lab operation seems unlikely unless its a mega jackpot, since you would want to work off the interest rather than drawing down the principal. Depending on the size of the award, though, you could fund several fellowships in addition to one’s own salary.
Running constant behavioral assays leaves a lot of time to work out these details.
LikeLike
April 24, 2013 at 10:20 am
Why I believe the stories that no one’s life was every made happy by winning the lottery:
Mrs. Grumble: What would you do if you won the lottery?
Prof. Grumble: Oh, that’s easy. I’d donate it to a foundation run by and for me, and I’d never have to worry about funding my research again.
Mrs.: [stares]
Prof.: Er, um, well …
LikeLike
April 24, 2013 at 10:45 am
I was just speaking to a program officer at NICHD. Their payline is 5th percentile for R01s
LikeLike
April 24, 2013 at 12:24 pm
@The Other Dave
Whatever you end up planning to do, remember that #7 is problematic because of the government’s rule that you cannot use federal (including R01) funds to pay someone to lobby for money from the government (which includes writing those R01s).
LikeLike
April 24, 2013 at 3:26 pm
@NatC: Oh, we have an understanding. They write the proposals before they are allowed to join the lab, or on their ‘own’ time once they join the lab.
I would of course also never pay someone to work on something not directly related to the currently funded proposal. Gosh and golly, that would also technically be mis-use of funds, wouldn’t it?
RAPE THE TAXPAYER! RAPE THEM!
My only regret is that I wasn’t smart enough to go into the defense industry. Or medical insurance. Or banking. Academia is for losers. Although economics doesn’t sound too bad. At least I wouldn’t have to worry about data or knowing how to use Excel properly.
…assuming there is a ‘proper’ way to use Excel. That program is retarded. If you use Excel for science, you are announcing that you are retarded. NO SERIOUSLY. I AM NOT JOKING.
I do like a nice 3D pie graph now and then, however. More NIH proposals should have 3D pie graphs, in my opinion. And clipart. There is not enough clipart in science proposals. Or papers. I would pay for color clipart in my figures.
LikeLike
April 24, 2013 at 3:51 pm
The Other Dave is providing a revealing glimpse into the stream-of-consciousness of all Third Reviewers everywhere.
LikeLike
April 24, 2013 at 5:19 pm
[write proposals] ‘or on their ‘own’ time once they join the lab’
…also not allowed.
LikeLike
April 25, 2013 at 4:32 am
That’s it TOD!! I’m putting clipart in my next proposal. I might at least draw a few chuckles.
LikeLike
April 25, 2013 at 4:34 am
…assuming there is a ‘proper’ way to use Excel. That program is retarded. If you use Excel for science, you are announcing that you are retarded. NO SERIOUSLY. I AM NOT JOKING.
Damn it. I always suspected that I was retarded. This confirms it.
LikeLike
April 25, 2013 at 11:36 am
I think the Other Dave is that crazed sorority emailer.
LikeLike
May 20, 2013 at 2:03 pm
I have a real problem with the idea that if the payline is X percent then all you need to do is submit Y applications to get funded. It just does not work that way. If the payline of your NIH institute is 10% (and many don’t publish paylines) then you need to write an application that is at least better than 90% of other applications.
The study sections that I have sat on have never looked at an applications and said “OK, so the payline for [NIH institute if your choice] is X% and this is [your name]’s Y application so we need to give him/her a score that is better than X%ile”. Study sections don’t know how many applications each investigator submits or the final paylines for institutes.
The best way to give yourself the fighting chance at funding is to identify the study section most appropriate to your work, and hopefully where there are people that know you and your work, and then submit the absolute best application you have ever written. And it does not hurt to get to know the appropriate PO and SRO assigned to your application, especially as you send in your A1.
LikeLike
May 20, 2013 at 4:15 pm
the idea that if the payline is X percent then all you need to do is submit Y applications to get funded.
Not what I meant at all and hopefully not what I said. It is more that random factors all have to align properly for you to make it over the fund line. You have to get the shake of the right reviewers, in the right mindset, with the right pile of other (shittier) applications in front of them. There is only one way to beat this game and that is to buy a lot of lottery tickets.
The best way to give yourself the fighting chance at funding is to identify the study section most appropriate to your work, and hopefully where there are people that know you and your work
This is undoubtedly the starting point. I have certainly secured the most triages and some of my best scores ever from the same study section. The former tells you that this is not enough. There are inevitable human factors having to do with appearing “too greedy”, the concept that “we just GAVE you one”, cultural factors having to do with certain types of application (say, risky versus workmanlike) and their biases about how a particular type of experiment “must” be performed. Not to mention their biases having to do with what aspect of Bunny Hopping is the best. And the fact that if you have more than one proposal in within a round, they are in direct competition with each other.
, and then submit the absolute best application you have ever written.
This is demonstrably nonsense. The “absolute best application” I’ve ever written is not funded. Probably the ones that did get funded don’t even line up as the next-best. I suspect that if I showed you a few dozen of my apps you’d be hard pressed to pick out the funded ones from the triages from the meh-percentile ones. Then the tales I can tell about the applications that have been reviewed well in sections that I’ve been on…
It is a huge mistake for applicants to let perfect be the enemy of pretty-damn-good.
And it does not hurt to get to know the appropriate PO and SRO assigned to your application, especially as you send in your A1.
POs are independent of study section, of course. And yes, it is a good idea to get to know POs and SROs. But why put your eggs in a single basket there either? Some SROs that I am friendliest with have seriously screwed me over on reviewer assignments and some have clipped off the bad reviewer just in time to save my ass. They have assignment issues, man, and the idea that they are going to be bailing your ass out just because you talk to them is insane.
LikeLike
May 20, 2013 at 4:56 pm
I think we should just say fuck it to the NIH and grovel at the feet of E Perlstein so he can help us crowdsource a mighty war chest of what $25k?
Which by the way where is he going to do his work and to whom and for how much does he make out the check for indirect costs on that money?
LikeLike
May 20, 2013 at 6:05 pm
I suspect that if I showed you a few dozen of my apps you’d be hard pressed to pick out the funded ones from the triages from the meh-percentile ones.
Dude, I bet I could pick out the dogs and lions!
LikeLike
May 21, 2013 at 4:41 am
Without RePORTEr, genius?
LikeLike
May 22, 2013 at 11:36 am
Having read your reply to my post I have to say that you sound amazingly bitter. How so? So some of your grants not been funded? Join the crowd. PO and/or SRO not bow and scrape at each of your emails/phone calls? They don’t actually work for YOU. Study sections not identifty your work as stellar? That makes it you….and everyone else.
You may view getting an NIH grant as a lottery. But that is simplistic because it means that each application has an equal chance at success. That sort of logic is insane. It does not explain why there those who have maintained funding for decades while there are others who have little to no success. When you work at a soft money institution, like I do, it does not pay to base your career on lottery tickets.
And, So from a statistical basis, you need to have put in 13 proposals to NIAID this year in order to have a fighting chance to get one does mean if the payline is X percent then all you need to do is submit Y applications to get funded.
LikeLike
May 22, 2013 at 12:38 pm
you sound amazingly bitter.
The word you are looking for is most likely “cynical”. Of the things that I am “bitter” about in my career, the way the NIH grant review game works is not one of them.
. But that is simplistic because it means that each application has an equal chance at success.
Of course it is a simplistic analogy. Applications do not have an equal chance in the sense that lottery tickets do. But neither are the chances lined up in strict accordance with some unified concept of omnipotent merit.
It does not explain why there those who have maintained funding for decades while there are others who have little to no success.
These decades that you refer to have seen success rates from established investigators change from damn near 50% to something under 20%. There is also a correlation of seniority with improved success that still is maintained. And yes, it is only recently that some of the most senior folks have finally come under the Lottery Rules that many others of us have had operating on our situation for years.
So what? I don’t blog for them. They are not my audience, for the most part, and stuff that works for them doesn’t work for the younger crowd.
When you work at a soft money institution, like I do, it does not pay to base your career on lottery tickets.
When you work in a soft money job category like I do, it is insanity itself to think that this NIH funding game is one you can survive in by submitting one “perfect” grant to the same study section every four years.
does mean if the payline is X percent then all you need to do is submit Y applications to get funded.
You have serious reading comprehension issues, my friend. “fighting chance” is in no way equal to “guarantee”.
LikeLike
May 29, 2013 at 8:44 am
Doesn’t matter how you want to slice and dice my comments, your bitterness just shines right through.
it is insanity itself to think that this NIH funding game is one you can survive in by submitting one “perfect” grant to the same study section every four years.
Well this is pretty much the approach that has worked for me for more than 20 years.
No, please don’t respond. It will only take time away from your next submission to your imaginary lottery.
LikeLike