Another white-hot discussion of NIH grant review over at Rock Talk

August 9, 2011

Whew, I can feel the keyboards screaming in agony as the OUTRAGED PI’s hammer away at their comments.
The Rock Talk blog entry is supposed to be about What Can You Do When Two Reviewers Contradict Each Other. Now admittedly, the blog advice is, well, laughable.

In this situation we encourage you to use your best judgment. Take a look at all of the reviewer’s comments and criterion scores* and the scientific review officer’s summary of the discussion and then make a decision on how best to proceed from there. If the summary statement is unclear, you can always contact your program officer for clarification.

ahahahahaa, no wonder people are pissed about that non-answer to the question they themselves have posed in the blog entry!


Anyway, you might as well read the discussion comments. The entire thread is filled up with people who
1) Can’t seem to grasp that two individual reviewers might have different opinions. I mean, whoever heard of scientists who disagreed over the interpretation of data or a finding or a hypothesis or theory, right? Sheesh. Anyway, clearly in grant review the only way two people could disagree is if they are incompetent or biased…going by these commenters.
2) Seem to think that review is only “fair” when they get a fundable score on their own proposal.
The only seeming voice of reason has garnered a bunch of “thumbs down” votes as of this writing.
Sigh.

Advertisements

25 Responses to “Another white-hot discussion of NIH grant review over at Rock Talk”


  1. Those reading that thread who actually want to do everything that is in their own power to get funded–rather than engage in unfounded conspiracy theories and refusal to accept the basic arithmetic of doing away with A2s–would do well to read Dr. Palmenberg’s comment that you link to. And then read it again. It accurately describes the real dynamic of reviewer and study section behavior and–unlike the “factual errors of incompetent and biased reviewers solicited by nefarious SROs unfairly killed my application” absurdity–is a useful guide to adjusting your applicant behavior to maximize the chance of getting funded.
    It is sad that these poor fuckes give “thumbs up” to delusional ranting that can’t help them get funded and “thumbs down” to the only comment on that whole thread with a connection to actual reality.

    Like

  2. Grumble Says:

    CPP characterizes the gist of the comments as, “’factual errors of incompetent and biased reviewers solicited by nefarious SROs unfairly killed my application’ absurdity.” However, the complaint of the original commenter, Holz, is legitimate, and it did not mention incompetence, bias, or the role of SROs. His complaint was simply that new criticisms were leveled in the A1 round, effectively tanking his application without allowing him the opportunity to respond to those new criticisms. The criticisms might have been eminently addressable, but that’s just too bad.
    Hence, subsequent commenters suggested that any new criticisms raised in the A1 round should open the door to an A2 round. That’s an interesting idea. Perhaps it would be difficult to implement, but one can imagine ways it could work.

    Like

  3. Neuro-conservative Says:

    How about 2 pages for Introduction to the Revised Application? Less burdensome than bringing back the A2, but perhaps allowing a slightly less arbitrary shot on A1.

    Like

  4. pinus Says:

    reading those comments is painful. the blame is flying everywhere…sro’s suck…PO’s don’t care….assistant professors are ruining review! One would think every idea is a good one based on those comments.

    Like

  5. DrugMonkey Says:

    I am particularly amused by people who describe themselves as having reviewed for ~25 years and relate sordid stories of clearly over-the-line bad behavior by SROs (and possibly themselves, see”I use back channels”).
    Why didn’t they raise a stink at the time???

    Like

  6. pinus Says:

    Because they were getting grants. It only ever becomes an issue worth raising your voice about if you don’t get paid.

    Like

  7. demiurge Says:

    We were in a similar situation where the first round reviewers were favorable but pointed out some gaps in our approach. The second round reviewers seemed to generally not understand our project all-together, although we addressed all the issues in the first round and had a much better proposal. They gave negative reviews with unhelpful comments. I don’t know how to fix it, but I’m sure this is a common problem: the reviewers of the first round would have loved our revised proposal. And this doesn’t even touch on contradictory reviewers in one review cycle.
    As it it, it seems like even if you write an excellent proposal, it’s up to chance if you are funded or not. That’s where a lot of the frustration in that thread is coming from.

    Like

  8. DrugMonkey Says:

    demiurge:
    Since they supposedly understand that it is up to “chance” why do they miss *why* this is so? I.e. There are genuinely too many excellent, not effectively differentiable, meritorious proposals. No, no, can’t be that. It simply must be underhanded bacKchannel dealings of craven NIH staff and incompetent reviewers colluding to rob them of their clearly deserved grant awards.
    This gets in the way of these people learning how to improve their odds.

    Like

  9. demiurge Says:

    It’s hard for people to admit when there is a catastrophe that there was nothing that could have been done. When a good research group with a good track record and good proposal ideas is disbanded because of lack of funding due to a single reviewer not understanding the grant, I can’t blame those affected for being angry at individual parts of the system as opposed to the system itself.
    I’m not naive enough to think that a change to any one part of the review process will fix this kind of problem. More funding to improve the total number of grants awarded will certainly help a lot, but there’s very little researchers can do to affect that. For many researchers I doubt that there is very much that they can do to “improve their odds” other than apply for funding sources other than NIH.

    Like

  10. DrugMonkey Says:

    Of course people can improve their odds. First and foremost by trying to understand bettEr how the system works, instead of whinging endlessly about craven POs, corrupt SROs and incompetent vengeful reviewers.

    Like

  11. DrugMonkey Says:

    Oh, and it is never the case that a single reviewer misunderstanding closes a “good” lab. If it really comes down to a single grant review, the PI is not doing her job. If you mean the last chance after a whole series of disappointing reviews, then you have to at least assign blame to several reviewers.

    Like


  12. When a good research group with a good track record and good proposal ideas is disbanded because of lack of funding due to a single reviewer not understanding the grant, I can’t blame those affected for being angry at individual parts of the system as opposed to the system itself.

    Any lab head that allows herself to get in the position of having the ongoing existence of her lab hinge on the review outcome of a single particular grant application is completely failing to do her fucken job. And yes, this includes relative greybeards who have run their labs on a single R01 for decades, and now are at risk of closure. Part of their job was to have paid attention to what was happening in the world of NIH funding over the last decade and to have realized that times had changed, and that they needed to diversify their grant portfolio to remain resistant to the whims of review panels.

    Like

  13. whimple Says:

    You saw the statistics that most labs are run off a single RO1 right? To insist PIs doing their job have multiple R01s is to deny reality. The existence of labs being at the whims of review panels is how academic science is done today.

    Like

  14. Cashmoney Says:

    You guys are missing the lede here! What’s all this talk about SROs pulling shenanigans with reviewer assignment? And, more importantly, how do I get in on that action??!!??!! I could totally use that. What’s with those back channels? How do I use those to squash my competitors?

    Like

  15. Grumble Says:

    “Any lab head that allows herself to get in the position of having the ongoing existence of her lab hinge on the review outcome of a single particular grant application is completely failing to do her fucken job.” -CPP
    Yes, but what is the quality of the science when scientists have to spend so much of their time writing grants? Essentially what you are saying is that any PI needs to constantly apply for grants, just get an occasional award to keep the lab afloat. I have managed to survive so far by submitting a constant stream of grants, but I have precious little time left for anything else. According to you, I’m not “failing to do my fucken job,” but according to me I am failing to do my fucken job because I don’t actually do science; I do fund-raising.

    Like

  16. anony Says:

    “According to you, I’m not “failing to do my fucken job,” but according to me I am failing to do my fucken job because I don’t actually do science; I do fund-raising.”
    Posted by: Grumble
    YES, YES, YES!

    Like

  17. DrugMonkey Says:

    If by “do science” you two mean benchwork (or your subfield equivalent), yeah, that isn’t the job of being a PI. You can dabble a bit but your real job is to set the conditions under which junior scientists can kick ass at the bench.

    Like

  18. Gruffi Gummi Says:

    It is a “non-answer” only in the context of NIH practices (triage). When ALL applications are discussed (e.g. at NSF) the panel develops a common stance, and the PI knows what needs to be improved/changed/addressed.
    Errors by single reviewers (or plain bullshits, which at NIH are routinely swept under the carpet) are largely eliminated when someone has to publicly defend his/her critique.

    Like

  19. DrugMonkey Says:

    I call bullshit on the notion that NSF approaches lead to universally uncomplaining PIs who feel they know what needs to be changed. Also that it always leads to a “common stance” on the panel.

    Like

  20. Neuropop Says:

    I agree that summary statements were mostly the reviewers’ justification for why
    they scored the proposal where they did and not what the applicant should do to improve it. This is true for NSF or NIH. As a reviewer, it is not my business to “fix” the proposal. As an applicant, I should not look for reviewers to do that for me. Rather I need to learn (or get help) to read the “tea-leaves”. Accusations of collusion, conspiracy or reviewer incompetence is all a poor excuse for not learning how to write proposals or not having had to do so (for the graybeards). Times are tough, and there are many good ideas out there. And yes, the PI’s job is fundraising. Except that fundraising occurs in the context of solid, novel and exciting ideas that keep the lab moving forward. Woe betide the PI whose lab depends on a single source of support.

    Like

  21. Gruffi Gummi Says:

    “Accusations of (…) reviewer incompetence is all a poor excuse for not learning how to write proposals”
    Fine. Explain this (just from ONE reviewer, in ONE critique):
    1. “SIGNIFICENCE/Weakness:
    It seems likely that a malevolent group engineering a disease vector would adopt as a first stratagem enabling it to resist the current antibiotics and this would largely bypass the stratagem being developed in this proposal.”
    2. In just the following paragraph:
    “APPROACH, Strengths
    This approach would be antibiotic independent.”
    Go ahead, defend this product of NIH peer review. IMO it has NOTHING to do with my “learning how to write a proposal”.

    Like

  22. Anonymous Says:

    When ALL applications are discussed (e.g. at NSF) the panel develops a common stance, and the PI knows what needs to be improved/changed/addressed.
    HAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHA!!!!!!!!!!!!!!!!!!!!!!!!

    Like

  23. Gruffi Gummi Says:

    “HAHAHAHAHAHA”
    Fine. In 50% of the cases. At NIH, this number for triaged applications is ZERO. NULL.
    Anyway, personally, I have never received any inconsistent/nonsensical critique from NSF. As for NIH, lo and behold, today I received my VERY FIRST Summary Statement that is free of bullshit of any kind. Not funded, based on: 70% priorities and perceived risks, 30% on scientific differences and probably (between the lines) that it’s the first round. I can live with this.

    Like

  24. Spiny Norman Says:

    “I have never received any inconsistent/nonsensical critique from NSF.”
    And anecdote is not the plural of data. I have received such a critique from NSF.
    Peer review is a process that involves human beings and is therefore inherently imperfect. Have a beer, get over yourself and write another fucking grant.

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: