EUREKA mechanism

May 2, 2007

The Medical Writing, Editing & Grantsmanship blog mentions a new trial grant mechanism, summarizing some main points:

the application itself, which is limited to 8 pages to explicitly address the significance/importance of the problem; the innovation/novelty of the hypothesis or methodology; the magnitude of the potential impact; and (something else new, borrowed, & blue) the size of the community affected. Even curiouser, the biographical sketch is limited to the 15 publications – the 5 most relevant, 5 most significant, and 5 most recent – plus a paragraph describing qualifications for the proposed research.
Preliminary data are allowed but not required.

Some additional details may be had from the NIH but I can’t find anything really official yet. There may be an announcement as early as this month.  [UPDATE 07/30/07: The initial announcement is out as an RFA] The alert reader will note that this proposal is formed to address some common criticisms of the current NIH R01 application…in spades.

First the “good”. Most of it IS good. R01 applications are far too lengthy for a number of reasons. Most importantly, I think the level of detail expected at present distracts from the central issues. Often times the review gets bogged down into a discussion of methodological minutia that has no place in grant review. (One, if we’re so concerned that this PI can’t figure out the basic methodology, why are we considering this person seriously as a PI. Two, if we are concerned that the control condition isn’t exactly correct to prove the point, isn’t this more appropriately the province of the paper review process? argh.) Shortening the application has to potential to head off much of what I consider unproductive aspects of review. The significance/impact/innovation part is all good, of course people are supposed to stress that currently and the top applications do- impact is uncertain. Finally, this is going to save a lot of PI time in grant preparation. Since there is a tremendous focus on methodological minutia this means that the grant itself has to be immaculate from a document perspective. Everything has to be consistent, timelines have to add up, the obligatory hypotheses better not be contradictory. (You know a bunch of “requirements” that have little to do with the way science is actually conducted!). Shorter and less detailed applications are going to save time and short circuit the “aha, hypothesis 1.A.II is slightly incompatible with hypothesis 4.C.IV! clearly the PI doesn’t know what s/he’s doing….Triage!” process…

The “bad”. Review bias. This is going to reinforce the bias for giving higher scores to well established researchers and lower scores to less experienced and transitioning investigators given the same objective quality of the proposal. The reason is that reviewers are concerned with issues of feasibility and likely productive outcome of the research. Particularly when it is viewed as “high risk” from a scientific standpoint, the notion of “success” of the project (meaning papers resulting) will be a concern. There is an entrenched belief that “track record” is highly important. There is a belief that a PI with a long career will be “able to get it done”, with a fairly nonspecific but nevertheless motivating belief that untried PIs will somehow blow it, waste the money, etc. I’ll likely get into this unsupported myth at some point but for now trust me, it is a powerful determinant of review outcome. The current long format R01-type proposal cannot completely cure the problem and indeed it doesn’t. However it gives the untried PI a fighting chance to address concerns. Lots of preliminary data, exquisitely argued/detailed research plans, additional autobiographical details slipped in (“sure these figures were from my postdoc days but i was running the project as evidenced by X, Y, Z….”), etc. All things that provide ammunition to the reviewer who is favorably disposed toward the application. Shorter applications are going to lessen the ability of the less-established investigator to establish a sense of confidence in the outcome of the project.

14 Responses to “EUREKA mechanism”

  1. bdf Says:

    Thanks for pointing me here; this clarifies a lot.
    But, regarding the methodological minutia: Isn’t the point simply to show that one can do these experiments and do them under the proper conditions?

    By knocking that part out, it seems, on the surface, as though you’d not be able to catch “rookie mistakes” that young/new investigators could still be making, nor bring back down to Earth some overly-optimistic researchers whose aims may be too lofty to be accomplished. I guess my point is that their aims need to be substantiated with evidence that they can be achieved experimentally, and this substantiation comes via detailed descriptions of one’s methods. You touched on this nicely in the “bad,” but it seems as though the “bad” outweighs the “good” here.

    Like

  2. bdf Says:

    PS I like your blog…can you tell? 🙂

    Like

  3. drugmonkey Says:

    Indeed the point of detailing experiments and methods is to show that one has a clue and give a chance to catch mistakes/problems in approach. Note it is not just “rookies” that make such mistakes. The problem is that this becomes an end in and of itself, to debate what are often not material differences in approach, to prejudge outcome of what are really empirical matters and can only be determined by the doing (catch-22 numero uno of grant review), to engage in what is really the paper review process (does this control work as intended? does this manipulation really show what you think it does?) and the like. Even to engage in what are really big-level scientific debates along the lines of I think X is going to happen / you think Y is going to happen. In the abstraction of science, each lab goes at it, publishes papers bashing each other and in the end we get to the essence of the matter. In the grant review process, it works to say “you can’t get funded to examine hypothesis X because I *think* that Y is so”…and so nobody does the experiment.

    It is not always so clear cut, true. But in my experience we often get too far down the road to discussing minor stuff instead of focusing on the big picture. So in this regard I favor shorter applications, to get away from this a bit.

    Your second point outlines the Stock Critique regarding “over ambitions proposal” quite nicely. You may perhaps mean it a little more in the sense of pie-in-the-sky proposals but I don’t really see too many of these. The question for proposals that run beyond what seems possible (technically or given the available data supporting the hypothesis) is whether they should be dinged merely for being ambitious. My view is that if there is enough described that is doable and is a significant contribution all by itself, then there should not be a subtraction for the pie/sky speculation or wishful experiments. Some people view this as a critique on how “realistic” the PI is. I think this is more a reflection of how well the PI understands grantsmanship and hope that ALL PIs have at least some pie/sky thoughts for the direction their research program is heading…

    Like

  4. PhysioProf Says:

    Other “bad”: My understanding is that the direct costs will be limited to $200,000 per year for three years, and (I think) no competing renewals. Seems to me that it would be much better to just write an R01, and not be so limited in money, time, and renewability. Is this really going to end up any better than the R21 mechanism?

    Like

  5. drugmonkey Says:

    From the RFA: “Support may be requested for up to $800,000 in direct costs (excluding consortium Facilities and Administrative [F&A] costs) over a four year period, prorated for shorter terms ($600,000 for three years, $400,000 for two years). Regardless of the term of support, direct costs (excluding consortium Facilities and Administrative [F&A] costs) may not exceed $250,000 in any one year. “

    It is better than R21 because it is more total money with up to 4 years of support instead of 2. Personally because of the way my research is usually configured, the change from 3 to 2 years of support nearly ruled out R21s for my work. Unfortunate because the ability to front-load costs (instead of uniform $100K per year) actually improved the fit to my projects. win some / lose some.

    With respect to the R01, yes, this is better. Mainly because they are trying very explicitly to stamp out some emergent properties of study sections that lead to a certain conservatism in review and an effective prevention of novel/exploratory stuff from getting funded. Naturally they could have done this by addressing the problem directly. But the creation of new mechanisms is an old NIH ploy to “fix” study section behavior.

    I’d suggest that the decades-long process of creating “new mechanisms” proves the error of the rationale. Study section behavior always gains the upper hand, the goal isn’t realized…and then Program has to create “new” approaches yet again.

    Like

  6. nosugrefneb Says:

    This EUREKA thing sounds so cool.

    Like

  7. PhysioProf Says:

    “I’d suggest that the decades-long process of creating ‘new mechanisms’ proves the error of the rationale. Study section behavior always gains the upper hand[.]”

    Yes, study sections are eeeevil. BWAHHAHAHAHAAHHAHAHAHAHAHAHAHAH!!!!!!!!!!!!!!!

    Just kidding. Seriously, let’s hope that some truly novel and risky research gets funded through EUREKA. I have a new proposal going in that could be EUREKAfied, but I am going to submit an R01. I guess I’m just not an early adopter.

    Like

  8. PhysioProf Says:

    This statistical analysis of NIH success rates as a function of constant-dollar NIH budget was just published:

    http://www.nature.com/ncb/journal/v9/n8/full/ncb0807-856.html

    The authors conclude that success rates are best when the budget is just past its trough.

    Like

  9. drugmonkey Says:

    of course. when people have been shelled out of the field and when people who can afford it have opted to wait “until things get better” those who do / have to put in applications are going to see reduced competition.

    Like

  10. PhysioProf Says:

    There is some fascinating stuff I found here:

    http://enhancing-peer-review.nih.gov/meetings/071807-summary.html

    This is a “meeting summary” by Dr. Larry Tabak and Dr. Jeremy Berg, Co-Chairs of the NIH Steering Committee Ad Hoc Working Group on Peer Review, of a meeting held on July 18. I found this assertion quite interesting:

    “[I]t is important that the research community recognize that NIH funding under current budget scenarios is more grant-in-aid as opposed to full project support for an integrated research program.”

    Somebody better tell that to the department chairs and deans of medical school basic science faculty.

    Like

  11. drugmonkey Says:

    All I can say is….FINALLY! Finally an assessment of the peer review process from NIH that makes some bloody sense. wow.

    good find Physioprof.

    any readers out there, go view that page…

    Like

  12. PhysioProf Says:

    After consultation with program, I have decided to give the EUREKA a shot.

    Like

  13. drugmonkey Says:

    Cool. Keep us posted, eh?

    Like

  14. PhysioProf Says:

    Of course.

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: