Please, Just Kill Me Instead

January 23, 2008

The Center for Scientific Review at NIH just released a two-page PDF pamphlet called the “Insider’s Guide to Peer Review For Applicants”. This publication contains little squibs written by former and current study section chairs intended as advice for applicants about how to write a good R01 research grant application. (h/t WriteEdit.)
Much of the advice is trite, but unobjectionable. However one of the suggestions relates to something that is one of PhysioProf’s bêtes noires.

Here is the suggestion that has me grinding my teeth:

Spend time on the Expected Outcomes, Data Interpretation, Pitfalls, and Contingencies section for each aim. The “expected outcomes” section shows you’ve got a logical strategy. The section on Data Interpretation gives insight into your depth of understanding the problem. The Pitfalls section shows how familiar you are with the proposed techniques and methodologies. Finally, in discussing alternative strategies, you can give us confidence you are able to deal with the problems that arise when experiments don’t work as expected.

This is so wrong, on so many levels, it makes me want to poke my eyes out with a red hot poker.
First, this is the kind of overly pedantic, clenched-sphincter approach to scientific research that is threatening to kill the NIH research enterprise, by driving out all of the biomedical scientists doing creative novel research with a high potential upside, and leaving behind only the bunumbed goldbrickers doing the same boring crap day after day, month after month, year after year. If you are proposing a research project that is designed to generate new information by exploring new areas with new techniques or model systems, then how the heck are you supposed to know what to expect, what might mess it up, and how to get around any problems that might arise?
The only kinds of studies in which you can really describe any of that crapola in detail without just pulling it randomly out of your ass are ones in which you basically already know to a high degree of certainty what the experiment is going to tell you. That sure isn’t the “crosscutting, groundbreaking, innovative” research that NIH says they are looking for. Sounds more like ditch-digging to me.
And “Data Interpretation”? You know what they’re talking about? “We will compare the measured values for the experimental and control groups using an unpaired t-test. If the p value if less than 0.05, then we will conclude that the null hypothesis that the two groups do not differ must be rejected. Alternatively, if the p value is greater than 0.05, we will conclude that the null hypothesis must be accepted.” Yeah, that’s a useful way to fill limited page space.
Second, what kind of people on a study section are going to be able to even slog through this treated sewage sludge, digest it, assess it, determine if it is “satisfactory”, and then give you a better score in light of it? Exactly, it’s the same kind of halfway-to-rigor-mortis boring twits that think science amenable to this kind of ass-backwards pre-performance analysis is the only kind worth doing, and that all the interesting science is a “fishing expedition”, “not hypothesis driven”, or “too speculative”.
Third, and finally, who in their right mind wants to write this kind of useless garbage? It is crap like this that makes writing a grant application even more painful than a root canal. And it makes it nearly impossible for the writer to get excited and stay excited for the sustained period of time it takes to write an R01. I would rather drive nails through my foot than write this kind of a grant application.
All this kind of overly-pedantic formulaic gotcha crapola is a corollary of the extremely detailed and long-winded 25 page application format, with junk like “In Specific Aim #3, sub-Aim (A)(c)(i)(2), we propose to use 50 mM NaCl, whereas in sub-Aim (A)(c)(i)(1), we propose to use 25 mM NaCl”. And guess where the 25 page application is going? Yep, right in the toilet.
So, the good news is that this “suggestion” by some superannuated ossified study section chair about “Expected Outcomes, Data Interpretation, Pitfalls, and Contingencies” is, soon enough, going to just be superfluous, rather than actively pernicious, because when the R01 application is 10 pages, there ain’t gonna be room for any of this verbal and scientific torture.
Instead, you’re gonna have to impress the study section with the novelty, creativity, and possibly groundbreaking nature of your proposed studies, and not with the fact that the research is so pedestrian that you already know exactly what is going to happen, or at least have some enumerated list of possible “pitfalls”.
Frankly, even with the 25 page R01, I have considered this experimental design kabuki to be an insult to my intelligence and creativity, and an insult to the intelligence and creativity of the members of the study sections that review my grant applications. I have written a substantial number of 25-page R01 applications that were well-liked by study sections, have had multiple R01s awarded, and I have never taken this “suggestion” to heart. It is just too painfully boring and, in my opinion, unnecessary if you take other steps (which we will get into in future posts here) to make it clear to a study section that they can feel confident that you will be able to handle these things as they arise in the course of the research even in the absence of all the sphincter-clenching kabuki suggested in the CSR pamphlet.

13 Responses to “Please, Just Kill Me Instead”

  1. Tree Fish Says:

    I whole-heartedly disagree with your objections. As you know, one of the key benefits of having an impressive pedigree is that peeps on the study section have (1) probably met you; (2) know one or more of your mentors; and/or (3) have reviewed your papers. That said, any of these alone or in combination work in the applicant’s favor because the study section can either ‘fill in the blanks’ regarding experimental design; or act as an advocate for the importance and cleverness of your proposed research.
    The schleps who lack that ‘impressive pedigree’ are not privy to such intangible support, and so are often vulnerable to the grumpy, skeptical eye of a fringe-insider that is jealous of the cleverness/novelty of the proposed research. As such, the proposal will be bandied about until it’s an -A2! The old study section saying is that a referee can only go to bat for 2 or 3 proposals each meeting; it helps if that referee has personal knowledge/linkage to the applicant. In the absence of such a connection, though, the applicant would be smart to treat the Exp Design section with the humble, thoughtful, and knowledgeable keypad to establish him/herself as a scientist with ‘insider’ talent, but an ‘outsider’ pedigree.
    I’ve been on both sides numerous times, so I wanted to chime in.
    Tree Fish


  2. apalazzo Says:

    Welcome to Sb!
    I was wasting time … ok … procrastinating (I am in seclusion writing ironically a k99) when I noticed your move. I’m slugging through section D right now. And as I type this the guilt is settling in. Looks like procrastination time is over … but I’ll check out the pamphlet, with a grain of salt.


  3. Drugmonkey Says:

    And allllllready we get into the descriptive versus prescriptive issue, good one PP!
    The comment dragged off of the “Insider’s Guide” is a defense of the validity of one of the most common StockCritiques. As the PP pointed out this is one of the worst features of the NIH grant process. I personally find this defense unconvincing but of course I was already a critic of this little “obligation” of grantsmanship.
    However, it IS a feature of grant review and a nearly inescapable on at present. PP’s experience notwithstanding.
    And it is most certainly the case the the more junior, less active or less well-known applicant will be held to a stricter standard on the “Expected Outcomes, Data Interpretation, Pitfalls, and Contingencies” front..


  4. Mark P Says:

    All I can say is, Ha! This is the inevitable result of the bureaucratization of any endeavor. We can’t evaluate the proposals so we resort to check-off lists.


  5. Beaker Says:

    PProf, you express my frustrations perfectly. I’ve been to the RO1 grant-writing classes, seminars, and tutorials. I’m sick of hearing about how I must anal-retentively write the pitfalls, alternatives, and data interpretation. I am, as you say, describing this crapola by puling it randomly out of my ass. I can think of 100 reasons an experimental won’t work, and 100 ways around the problem. Picking and choosing is random. I feel like I am doing a book report to get an “A” in Jr. High.
    That’s not how good science is done. If we knew what was gonna happen, it wouldn’t be science. Throughout my career until now, I’ve succeeded by following my nose. If the train is leaving, I will get in and ride. If the boat is sinking, I will swim, not drown. In my career, none of the successful studies that led to papers were anticipated ahead of time in grant applications. That’s 35 papers of “fishing expeditions.” We succeeded because we were asking important questions, and created feasible experiments to address important problems.
    I have a colleague (who is well-funded) who breaks his experimental section into “tasks to be accomplished.” I NEVER want to be like that, for it implies that simply by following a “to do” laundry list of pre-designed experiments, I will make a significant scientific discovery.
    So I raise the following question: If I don’t fill my 15-pages of 11-point Ariel with a bunch of “pitfalls and alternatives” crapola, what goes in there instead? What do you put in there?


  6. juniorprof Says:

    PP, I completely agree… I really enjoy writing grants (love it in fact) but I dread these contingency-type sections. I fail to see how they demonstrate that you are a quality scientist and I, personally, find them distracting. They dilute ideas and break up the flow of grants that might otherwise tell a nice-concise story. I do not, on the other hand, yet have the intestinal fortitude to leave them out.
    So what to do about it? Will cutting page limits solve the problem? What are the chances that NIH will cut R01/21 page limits?


  7. Drugmonkey Says:

    “Will cutting page limits solve the problem?”
    Over time, probably. A greater fraction of applicants will be dropping all the “discussion of outcomes and alternatives” business which will defang the StockCritique. A greater fraction of reviewers will be familiarizing themselves with this practice as well. You, DearReader, will be preaching the gospel on this topic to your local study section participants (right?). However I anticipate a nasty interval of transition….
    “What are the chances that NIH will cut R01/21 page limits?”
    This is a done deal as far as I am concerned. They are so far down the road on this in terms of hot air and the EUREKA grant structure that I cannot imagine some version of cut-the-length not appearing.


  8. Luna_the_cat Says:

    juniorprof says:

    …I really enjoy writing grants (love it in fact) …

    What I want to know is, what medications are you on, and where can I get some?


  9. juniorprof Says:

    No meds, just diet pepsi. Seriously though, I have problems staying focused. This has always served me well in coming up with new ideas but is terrible for putting a plan together to rigorously test a set of hypotheses on a general idea. Writing grants helps me do this… I suppose its a defense mechanism, but one that allows me to overcome a glaring weakness. I have to admit, I like the competition too.


  10. Monado Says:

    I agree it sounds dreary. Much better would have been, “_Think about_ expected outcome, etc.” but if they’re not doing that, why experiment? “Make some notes about”? “Don’t forget”? It can be hard to strike the right note.
    Welcome to Scienceblogs!


  11. anne Says:

    I totally agree with PP in his/her assessment of what I thought was dead (after almost 3 years of discussion). It sounds as though that “old stuff” is still alive or perhaps is that “resurrection” can happen, can it ?.
    I also thought that the page limit was something done. D-O-N-E !!!!.


  12. anonymous Says:

    Maybe a fresh, more explicit new powerpoint is needed !!!!.
    Have you seen the movie ” Nine”. I bet you haven’t. Otherwise, I would have heard you already singing, ” My friend makes powerpoints”,
    instead of ” My husband makes movies” (of course).


  13. whimple Says:

    “Alternatively, if the p value is greater than 0.05, we will conclude that the null hypothesis must be accepted.”
    This is not a good idea. 🙂


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: