A Penalty Box for Bad Grant Writers

March 27, 2009

Prof-like Substance picked up on something I’ve been neglecting to cover after a correspondent* sent me the link. A policy from a funding agency which is intended to cut down the peer review burden. A policy with very nasty implications.
Prof-like Substance wondered what would happen if the NSF adopted a similar policy:

if NSF were to put this policy in place it would completely change the way I apply for funding. As of right now, I have two different grants under consideration at NSF and I plan to submit another in July. I submitted my first one last July, before I arrived here, and that one was not funded and did not receive very high ratings because of the lack of preliminary data. If the resubmitted version of this grant (submitted in January) and the most recent grant that just went in (two weeks ago) were to be similarly ranked, I would essentially be shut out from funding for the following two years. Rather than getting feedback on different proposals that as a new investigator, I would instead either have to change my research focus and apply to a different agency (a scenario where having preliminary data would be unlikely) or be looking at shutting down the lab and finding work elsewhere.


sooo, what is the The Engineering and Physical Sciences Research Council (EPSRC) up to?
Nature (also see):

The EPSRC says that scientists will not be allowed to apply for research funding for 12 months if, in the past 2 years, they have had three or more proposals ranked in the bottom half of a funding prioritization list, and also have less than 25% of all their proposals funded in that time.

Science:

EPSRC is the largest U.K. funder of engineers and physical scientists, handing out grants totaling more than £475 million in 2007-08 to more than 3200 researchers….After rejecting options such as charging for proposal submissions or placing quotas on institutions, on 12 March the council announced a new policy …EPSRC notes that this policy should exclude about 200 to 250 people and is retroactive: Letters to those excluded go out on 1 April, and their proposals won’t be considered after 1 June…Reid says the idea is to weed out the small number of scientists who submit multiple, poor applications; the estimated 5% excluded submit 10% of applications, says EPSRC. “They’re operating a scattergun approach” and placing a “huge burden” on the peer-review process, Reid says. He adds that EPSRC consulted with university officials who expressed a desire to know which researchers need mentoring to help them obtain funding.

time-out.jpgSimple question: If this policy had been in place at the NIH would it have affected you at any time in the past or present, DearReader?
I am going to have to go back and crunch my grant numbers because I bet it would have applied to me at some point. I for sure have amassed “3 or more” proposals with triages within a 2-year interval. I’ll have to check on the overall 25% thing but I bet I would have qualified at some point for the penalty box if NIH was doing this nonsense.
…and yeah, in my job type a 1-year ban from even submitting would likely have put me out of my career.
Go play over at PLS….
__
*PP

No Responses Yet to “A Penalty Box for Bad Grant Writers”

  1. Orac Says:

    I don’t think it would have effected me for the simple reason that I got lucky on two Army grants early in my career, after which I submitted my first R01, which got triaged, and then ended up waiting over two years before submitting it as a new R01, which was funded.
    This year, however, I’m submitting multiple Challenge Grants, two R21s, my competitive renewal for my R01, and, if I can pull it off, I’m finishing it all up by submitting a brand new R01 on another topic by February or June next year. (Surgeons are gluttons for punishment.) That’s at least six shots at NIH-funded grants; I could quite possibly end up with three or more of them triaged (hopefully not my R01 competitive renewal!) in today’s climate.

    Like

  2. Eric Lund Says:

    Many NSF programs already have success rates below 25%. In many fields, half or more of the scientists would be sent to the penalty box, which is bad news for anybody on soft money.

    Like

  3. JohnV Says:

    Am I thinking about this wrong? It seems like with a funding rate as low as it is, a requirement for people to have at least 25% of their grants funded would result in a lot of people receiving 2 year bans?
    I guess the smart strategy would be to figure out some way to hold off on submitting proposals for a year or two, then when most of your colleagues have been black listed you can sneak in? 😛

    Like

  4. DrugMonkey Says:

    It seems like with a funding rate as low as it is, a requirement for people to have at least 25% of their grants funded would result in a lot of people receiving 2 year bans?
    Unless you had a big “N” of performance behind you to insulate you against a temporary downturn. I joke all the time about outrage coming from people at the end of their careers who are having to revise a grant application for the first time ever. It is not an entirely ridiculous joke, or it wasn’t about 5 years ago anyway. Also a similar joke about people getting triaged for the first time EVAH! aieeee!
    Youngsters that got their start within the past 10 yrs may not be that familiar with just how good things were for many of those who we consider luminaries. Because of course, they were the cream of the crop for the large part of their careers. So things were pretty easy when it came to getting the grants.
    The difference now is that nobody is totally immune to triage. and those people that used to be essentially immune to getting triaged got some funny ideas in their heads..

    Like

  5. JohnV Says:

    To be honest, given my short time as a scientist (8 years counting grad school) I didn’t even realize people ever had grants funded without revision 😦

    Like

  6. drdrA Says:

    The ugly truth is that there was a high rate of triage recently many times just because you had a first submission…and weren’t yet well versed in grant-speak.
    So, now let’s say you are a new assistant professor, and take all advice given to you on submitting a grant for every deadline in a given year (we’ll just imagine that there are three deadlines in a given year)… let’s just say you do this in your second year. All three first submissions get triaged. …
    boom, dead for submitting for two years.
    Dead career in academic science.
    Stupid plan on the part of the funding agency.

    Like

  7. DrugMonkey Says:

    Don’t take my word for it JohnV. Assuming you are talking about NIH, go to CRISP and do wild-card searches for “1R01%”. Gate by your favorite IC and jump back in 5 year steps. Check out the proportion of new grants that are funded revised/unrevised….

    Like

  8. drdrA Says:

    And I should have added, that is not a penalty box for new investigators… that is a graveyard.

    Like

  9. Hap Says:

    Considering the worry over the increasing average ages of those receiving first-time grants (at least for NIH, but I didn’t figure they were the only ones worried), wouldn’t this be a highly counterproductive policy if implemented here?
    “If you aren’t the absolute best in your field or you make a mistake in your grant proposals, don’t even bother trying to be a research professor” is what it seems to say. Which might get rid of that pesky oversupply of scientists thing in the long run and the overemployment problems of scientists in the private sector in the medium to long terms, if those were problems they were actually looking to solve. If not, it seems rather counterproductive, like blowing your leg off with a mine because you have knee problems.

    Like


  10. You’re all misunderstanding. You have to have 25% or less of your grants funded AND be in the bottom 50%ile of applications with three or more proposals.

    Like

  11. DrugMonkey Says:

    You have to have 25% or less of your grants funded AND be in the bottom 50%ile of applications with three or more proposals.
    Yeah and this ain’t that hard to trigger in the NIH system of the past several years, is what I’m saying. Having 3 triages in a 2-year span is easy to accumulate. I have for sure in the past had intervals in which this was true.
    If you count every submission (original or revised) as an independent event (which is how the NIH views it mostly) I would argue it is also very easy to trigger the 25% criterion.
    …or maybe I just suck.
    Another implication for strategy arises. I’ve abandoned more than one application before I expended my allowed revisions, even in some cases where things were somewhat promising. Why? Because I had a better one and/or a grant about to be funded and didn’t really need any more at the time. If batting percentage were to be important, you’d have to take all these opportunities to the mat and try to get them funded anyway.

    Like

  12. becca Says:

    Does NIH even need it? Are there enough absolutely heinous proposals out there that reviewers feel this is a major source of the burden of reviewing?

    Like

  13. DrugMonkey Says:

    Does NIH even need it? Are there enough absolutely heinous proposals out there that reviewers feel this is a major source of the burden of reviewing?
    if someone was spamming crap across a large number of study sections this would be invisible to individual reviewers seeing only one or two.

    Like

  14. perceval Says:

    Weighing in as somebody who actually submits to EPSRC. The mechanism is very different from the NIH. Most bread-and-butter research grants are for 3-3.5 years, with 3 years as the most common length. Grants cannot be renewed; you have to write and submit a completely new application. Incremental work doesn’t cut it.
    So imagine that success rates are 10-25%, you need quite a few of those three-year grants to keep your lab staffed and running (roughly 1 per postdoc), and only full academic staff can be PIs. If you have 2-3 good postdocs / researchers that you want to keep around, you need to write (or have them write) 4-6 grants to stand a chance of keeping at least one of them. You’re always PI, of course. 1-2 duds per year and that’s you. For perspective, personally, I have two EPSRC grants with council (researcher co-investigator on both, different PIs), with a third potentially in preparation plus grant applications to three other funding sources in the works. You get the idea. So far, I’ve had two highly-rated EPSRC grants rejected (both times researcher co-I).
    All you have to describe your brilliant idea is SIX pages INCLUDING references. There’s extra for justification of resources and track record and two extra pages of friggin’ impact statement, but still, six pages is not a lot if you want to make sure your references are exhaustive and you don’t hurt a gentle reviewer’s soul by not citing them (forsooth!)
    If your brilliant idea didn’t get funding, you may not resubmit it, unless explicitly invited to do so. That may sound reasonable, but look at how EPSRC decides. From what DrugMonkey is saying, your NIH has study sections that are quite specific to a research area, right? Well, EPSRC has recently moved to a new structure where all Healthcare Engineering is judged at the same panel, all Information and Communication Technology (=computer science) grants are judged at the same panel, etc. It’s like having one panel for all of microbiology, one for all of biochemistry, one for organic chemistry and so on and so forth. Since panels are far less specialised, dud reviewers are more likely to really pull your proposal down, because it’s likely that NOBODY on your panel will understand your specific subarea of, say, computer science. This whole move to single, broad panels has been devastating for the kind of interdisciplinary work that our research centre engages in.
    So, Comrade PhysioProf, although some of my colleagues share your view, many of them are up in arms.

    Like

  15. May Day Says:

    Another point to consider is that this is being applied retroactively. This sort of policy may well require people to rethink their grant strategies, but that doesn’t help people who are caught in the trap for grants submitted prior to this.
    I’m also particularly concerned about the implications for new investigators. One of the main themes of this blog is that grantsmanship is a skill like any other, and to me it’s a skill that can really only be improved by practise and going through the process. So if new investigators end up being discouraged to submit grant applications, how do they gain the skills to become ye big olde cheese?

    Like

  16. qaz Says:

    Yikes! This policy would have completely destroyed my career. It took me years (>5) to figure out how the grant system worked. This policy is clearly targeted at people who are spamming the system, but it also is going to nail people who are learning. This policy would have absolutely destroyed my career (and the careers of every junior scientist in my field that I can think of off hand).
    Note also how this really lets the politics of study section destroy careers. As has been discussed many times on this blog, we all know how hard it is to get tiger hopping or bunny crawling past a bunny hopping study section. Imagine if the bunny hoppers could force you into time-out by triaging your grant twice!
    By the way, does EPSRC have any evidence that people are actually spamming their system? Or is this just a misguided policy pulled out of some bureaucrat’s a**?

    Like

  17. Ted Says:

    I think this policy is heading in the wrong direction. I would hate to see the science community miss out on a great idea that should be funded just because the person had a couple that ‘were unworthy’.

    Like

  18. Lorax Says:

    Well I should be taken out back and flogged because I suck.
    First 3 R01s (first, + A1 and A2) two scores > 25%, last not funded. Went in as a new grant 4%…WOOT
    Second grant 3 R01s (as above) two scores > 25%, last not funded. Went in as new grant….triaged.
    R21 this year triaged.
    So I am 1 for 7 submissions with a challenge a new R01 and an A1 coming up. I will likely (though I hope not) be batting less than 10% success rate.

    Like

  19. Lorax Says:

    BTW we have all had reviews (Im assuming) where the reviewers are killing us for trivial details “Didnt explain how they were doing a Southern blot” or, as far as I can see, malevolent reasons “Completed major portions of aim 2 since the first submission and has now replaced it with new studies that we have not yet had a chance to fix”
    Yes, lets punish the submitters for the potential failings of reviewers (I frequently see people on study sections whose work is AA ball at best and seldom see the major leaguers in my field serving on study sections). This is a lousy policy being proposed.

    Like

  20. Propter Doc Says:

    The other problem with this policy concerns the ‘bottom half of submissions’ concept. This will, in fact create more work for reviewers and those on panels because they only used to care about the top 20% or so that had a chance in hell of being funded. Now the 50% point will be a critical point and will require a little more consideration.
    Either way, this proposal screws British academics, particularly young academics. Yet again the UK funding system finds a way to hand money to large established groups in the top universities at the expense of everyone else. The short sighteness being displayed in UK funding policy currently is mind blowing.

    Like

  21. Propter Doc Says:

    Just to follow up – I’ve not heard any discussion of academics ‘spamming’ the system with applications.
    A number of universities in the UK currently have ‘research quotas’ in place for academic staff. One ugly feature is that the academics must be seen to be applying for funding. Grant rejected? Well, it ticks the university box just to resubmit without any revision. I think this is one of the problems this dumb-ass policy seeks to address.
    The other ugly features of these quotas include a minimum number of grad students/year, a minimum income over 5 years and a minimum number of publications per year.

    Like

  22. whimple Says:

    Science doesn’t work well on a quota system. The UK is just ensuring mediocrity this way.

    Like

  23. msphd Says:

    Sounds like a very effective way of cutting the number of scientists very quickly.
    Also sounds like it will select for better salespeople who can communicate to a broader audience, or who have political connections with the reviewers (if it’s anything like the NIH is now).
    I agree with some of the other commenters that it’s unfair to punish people who have very little time to try again before they are excluded (2 months?).
    However, this kind of policy might also produce a much-needed pressure on “training” programs to actually TEACH grantwriting (DM- I prefer the non-gendered word to “grantsmanship”). I doubt that’s an intended consequence, but it could be a good one.
    So overall, I’m curious to see if they keep it. As policies go, usually nothing is set in stone. If there’s enough of a backlash, it might not last.
    But, it might not be an entirely bad thing. I’d be curious to know if, demographically speaking, there are any patterns emerging among the group using the shotgun approach- do we know if they are mostly new investigators who would be excluded?
    Is it possible that they are mostly old guys who need to retire anyway, and this is actually a really sneaky way to force them out?
    I can’t believe nobody… well, I can. It’s possible nobody did that calculation to see who is on the (first) List of Exiles before they drew up the policy. But I somehow doubt it.

    Like

  24. Propter Doc Says:

    I think we can safely say it is a bad thing. It is a bad precedent, it is a bad idea, it is just bad. It is not a backhanded way to remove the upper tier of UK academia that are somehow preventing young scientists getting their shot at the job.
    And I’m sick and tired of hearing that the solution for all of these problems is more bloody teaching and more bloody professional development. You can teach all the grantmanship you like, but if you’re idea is not appropriate for the call (and I appreciate that is part of grantmanship, to an extent), or if you’re ‘taught’ by someone who doesn’t play the game quite right, it is a waste of bloody time.
    The solution to these problems is to consult fully with the community that applies for these grants, to identify what the cause of ‘spamming’ grants is, and to come up with a solution that does not adversely affect younger academics or those not working in the top 10 deparments with all their nepotistic gloryl.

    Like

  25. JD Says:

    It seems unlikely that this policy is going to impact senior investigators disproportionately. For that to be the case, we would need to see a lot of senior people who have trouble getting funding.
    The main abuse that it might stop is strategies like making junior PhD student write lots of grants in the name of the PI. That’d be pleasant.
    But it also means that a junior person needs to be vwry successful right from the outset to have a career. A few mistakes in things like “study section” could end up being fatal. I am in a subfield where hard money is effectively gone so it is “write or die” if I ever become a junior faculty member. This makes it easy to see how things could go wrong with “one submission per deadline” . . .

    Like

  26. whimple Says:

    The solution is to get rid of tenure altogether. That way the universities can just fire people who can’t compete for grants, which will automatically take care of the problem.

    Like

  27. Hap Says:

    I’m sure it would – why the hell would people take low-paying jobs with lots of uncertainty and difficult funding? Oh, and if they come up with ideas to enhance their low pay, the U will probably take a big cut. If people want to follow their own ideas, at that point they might be better starting their own companies.
    It’ll probably solve the problems of scientist overpopulation and overemployment. I’m not certain those were problems anyone was looking to solve, though.

    Like


  28. The solution is to get rid of tenure altogether.

    Grabs popcorn. Takes seat.

    Like

  29. qaz Says:

    Whimple #26, you should mark sarcasm with a smiley-face. (That’s why it was invented [because someone made a joke at MIT about mercury in the elevator and no one realized it was sarcastic.]) Since there’s no sarcasm-mark, I’ll take the bait.

    The solution is to get rid of tenure altogether.

    Huh? Is your goal to kill science? Or just to drive it underground?
    We have enough problems with the current “translational” fashion that science is only good if it serves immediate needs. (Something we all know to be wrong, but seemingly can’t convince the NIH system.) The reason we have federally funded research is that science takes 30 years to come to fruition. No company is going to do research that will help the entire world in 30 years. Forcing faculty to chase immediate grants will put us all in that same boat.
    What tenure does is it explicitly decouples science from immediate grant/fashion needs. Sure, there’s some eventual deadwood, but in my observation, the number of faculty pursuing crazy ideas that pan out far outnumber the deadwood. Tenure allows scientists to pursue goals independently of grants. (Yes, I know it’s hard to run a big biolab without grants, but there’s science that can still be done without grants. More importantly, tenure allows one the freedom to chase something for the 3-5 years one needs to before one can get the grant.)
    I should also point out that current faculty salaries are extremely low relative to what any of these people could easily make. The opportunity to pursue long-term goals without fear of reprisal is one of the major perks that draw top talent to science.

    Like

  30. Cashmoney Says:

    Oh c’mon qaz, tenure isn’t necessary for non-deadwood which begs a certain question.

    Like

  31. qaz Says:

    Oh c’mon qaz, tenure isn’t necessary for non-deadwood which begs a certain question.

    I’m not sure what the “certain question” is, but I know of a number of examples where people have gone out on pretty risky limbs after tenure that they didn’t dare go out on before tenure. (And some of them paid off.) Whether tenure was necessary or not, I don’t know, but I know that they perceived it as such. [Discussion continued in response to DM’s next post.]

    Like


  32. […] we last discussed the UK Engineering and Physical Sciences Research Council (EPSRC) they had just instituted a […]

    Like


Leave a comment