Another white-hot discussion of NIH grant review over at Rock Talk
August 9, 2011
Whew, I can feel the keyboards screaming in agony as the OUTRAGED PI’s hammer away at their comments.
The Rock Talk blog entry is supposed to be about What Can You Do When Two Reviewers Contradict Each Other. Now admittedly, the blog advice is, well, laughable.
In this situation we encourage you to use your best judgment. Take a look at all of the reviewer’s comments and criterion scores* and the scientific review officer’s summary of the discussion and then make a decision on how best to proceed from there. If the summary statement is unclear, you can always contact your program officer for clarification.
ahahahahaa, no wonder people are pissed about that non-answer to the question they themselves have posed in the blog entry!
Anyway, you might as well read the discussion comments. The entire thread is filled up with people who
1) Can’t seem to grasp that two individual reviewers might have different opinions. I mean, whoever heard of scientists who disagreed over the interpretation of data or a finding or a hypothesis or theory, right? Sheesh. Anyway, clearly in grant review the only way two people could disagree is if they are incompetent or biased…going by these commenters.
2) Seem to think that review is only “fair” when they get a fundable score on their own proposal.
The only seeming voice of reason has garnered a bunch of “thumbs down” votes as of this writing.
Sigh.
DrugMonkey 
August 9, 2011 at 7:21 pm
Those reading that thread who actually want to do everything that is in their own power to get funded–rather than engage in unfounded conspiracy theories and refusal to accept the basic arithmetic of doing away with A2s–would do well to read Dr. Palmenberg’s comment that you link to. And then read it again. It accurately describes the real dynamic of reviewer and study section behavior and–unlike the “factual errors of incompetent and biased reviewers solicited by nefarious SROs unfairly killed my application” absurdity–is a useful guide to adjusting your applicant behavior to maximize the chance of getting funded.
It is sad that these poor fuckes give “thumbs up” to delusional ranting that can’t help them get funded and “thumbs down” to the only comment on that whole thread with a connection to actual reality.
LikeLike
August 9, 2011 at 10:38 pm
CPP characterizes the gist of the comments as, “’factual errors of incompetent and biased reviewers solicited by nefarious SROs unfairly killed my application’ absurdity.” However, the complaint of the original commenter, Holz, is legitimate, and it did not mention incompetence, bias, or the role of SROs. His complaint was simply that new criticisms were leveled in the A1 round, effectively tanking his application without allowing him the opportunity to respond to those new criticisms. The criticisms might have been eminently addressable, but that’s just too bad.
Hence, subsequent commenters suggested that any new criticisms raised in the A1 round should open the door to an A2 round. That’s an interesting idea. Perhaps it would be difficult to implement, but one can imagine ways it could work.
LikeLike
August 10, 2011 at 12:56 am
How about 2 pages for Introduction to the Revised Application? Less burdensome than bringing back the A2, but perhaps allowing a slightly less arbitrary shot on A1.
LikeLike
August 10, 2011 at 7:37 am
reading those comments is painful. the blame is flying everywhere…sro’s suck…PO’s don’t care….assistant professors are ruining review! One would think every idea is a good one based on those comments.
LikeLike
August 10, 2011 at 10:57 am
I am particularly amused by people who describe themselves as having reviewed for ~25 years and relate sordid stories of clearly over-the-line bad behavior by SROs (and possibly themselves, see”I use back channels”).
Why didn’t they raise a stink at the time???
LikeLike
August 10, 2011 at 11:23 am
Because they were getting grants. It only ever becomes an issue worth raising your voice about if you don’t get paid.
LikeLike
August 12, 2011 at 11:01 am
We were in a similar situation where the first round reviewers were favorable but pointed out some gaps in our approach. The second round reviewers seemed to generally not understand our project all-together, although we addressed all the issues in the first round and had a much better proposal. They gave negative reviews with unhelpful comments. I don’t know how to fix it, but I’m sure this is a common problem: the reviewers of the first round would have loved our revised proposal. And this doesn’t even touch on contradictory reviewers in one review cycle.
As it it, it seems like even if you write an excellent proposal, it’s up to chance if you are funded or not. That’s where a lot of the frustration in that thread is coming from.
LikeLike
August 12, 2011 at 11:19 am
demiurge:
Since they supposedly understand that it is up to “chance” why do they miss *why* this is so? I.e. There are genuinely too many excellent, not effectively differentiable, meritorious proposals. No, no, can’t be that. It simply must be underhanded bacKchannel dealings of craven NIH staff and incompetent reviewers colluding to rob them of their clearly deserved grant awards.
This gets in the way of these people learning how to improve their odds.
LikeLike
August 12, 2011 at 2:30 pm
It’s hard for people to admit when there is a catastrophe that there was nothing that could have been done. When a good research group with a good track record and good proposal ideas is disbanded because of lack of funding due to a single reviewer not understanding the grant, I can’t blame those affected for being angry at individual parts of the system as opposed to the system itself.
I’m not naive enough to think that a change to any one part of the review process will fix this kind of problem. More funding to improve the total number of grants awarded will certainly help a lot, but there’s very little researchers can do to affect that. For many researchers I doubt that there is very much that they can do to “improve their odds” other than apply for funding sources other than NIH.
LikeLike
August 12, 2011 at 2:59 pm
Of course people can improve their odds. First and foremost by trying to understand bettEr how the system works, instead of whinging endlessly about craven POs, corrupt SROs and incompetent vengeful reviewers.
LikeLike
August 12, 2011 at 3:11 pm
Oh, and it is never the case that a single reviewer misunderstanding closes a “good” lab. If it really comes down to a single grant review, the PI is not doing her job. If you mean the last chance after a whole series of disappointing reviews, then you have to at least assign blame to several reviewers.
LikeLike
August 12, 2011 at 4:58 pm
Any lab head that allows herself to get in the position of having the ongoing existence of her lab hinge on the review outcome of a single particular grant application is completely failing to do her fucken job. And yes, this includes relative greybeards who have run their labs on a single R01 for decades, and now are at risk of closure. Part of their job was to have paid attention to what was happening in the world of NIH funding over the last decade and to have realized that times had changed, and that they needed to diversify their grant portfolio to remain resistant to the whims of review panels.
LikeLike
August 12, 2011 at 7:33 pm
You saw the statistics that most labs are run off a single RO1 right? To insist PIs doing their job have multiple R01s is to deny reality. The existence of labs being at the whims of review panels is how academic science is done today.
LikeLike
August 12, 2011 at 8:47 pm
You guys are missing the lede here! What’s all this talk about SROs pulling shenanigans with reviewer assignment? And, more importantly, how do I get in on that action??!!??!! I could totally use that. What’s with those back channels? How do I use those to squash my competitors?
LikeLike
August 13, 2011 at 11:47 pm
“Any lab head that allows herself to get in the position of having the ongoing existence of her lab hinge on the review outcome of a single particular grant application is completely failing to do her fucken job.” -CPP
Yes, but what is the quality of the science when scientists have to spend so much of their time writing grants? Essentially what you are saying is that any PI needs to constantly apply for grants, just get an occasional award to keep the lab afloat. I have managed to survive so far by submitting a constant stream of grants, but I have precious little time left for anything else. According to you, I’m not “failing to do my fucken job,” but according to me I am failing to do my fucken job because I don’t actually do science; I do fund-raising.
LikeLike
August 14, 2011 at 8:04 am
“According to you, I’m not “failing to do my fucken job,” but according to me I am failing to do my fucken job because I don’t actually do science; I do fund-raising.”
Posted by: Grumble
YES, YES, YES!
LikeLike
August 14, 2011 at 10:27 am
If by “do science” you two mean benchwork (or your subfield equivalent), yeah, that isn’t the job of being a PI. You can dabble a bit but your real job is to set the conditions under which junior scientists can kick ass at the bench.
LikeLike
August 14, 2011 at 10:33 am
Grumble and anony, this older post is related to my point
http://scienceblogs.com/drugmonkey/2010/05/assistant_professors_who_canno.php
LikeLike
August 15, 2011 at 3:09 pm
It is a “non-answer” only in the context of NIH practices (triage). When ALL applications are discussed (e.g. at NSF) the panel develops a common stance, and the PI knows what needs to be improved/changed/addressed.
Errors by single reviewers (or plain bullshits, which at NIH are routinely swept under the carpet) are largely eliminated when someone has to publicly defend his/her critique.
LikeLike
August 15, 2011 at 6:52 pm
I call bullshit on the notion that NSF approaches lead to universally uncomplaining PIs who feel they know what needs to be changed. Also that it always leads to a “common stance” on the panel.
LikeLike
August 16, 2011 at 11:56 am
I agree that summary statements were mostly the reviewers’ justification for why
they scored the proposal where they did and not what the applicant should do to improve it. This is true for NSF or NIH. As a reviewer, it is not my business to “fix” the proposal. As an applicant, I should not look for reviewers to do that for me. Rather I need to learn (or get help) to read the “tea-leaves”. Accusations of collusion, conspiracy or reviewer incompetence is all a poor excuse for not learning how to write proposals or not having had to do so (for the graybeards). Times are tough, and there are many good ideas out there. And yes, the PI’s job is fundraising. Except that fundraising occurs in the context of solid, novel and exciting ideas that keep the lab moving forward. Woe betide the PI whose lab depends on a single source of support.
LikeLike
August 16, 2011 at 2:35 pm
“Accusations of (…) reviewer incompetence is all a poor excuse for not learning how to write proposals”
Fine. Explain this (just from ONE reviewer, in ONE critique):
1. “SIGNIFICENCE/Weakness:
It seems likely that a malevolent group engineering a disease vector would adopt as a first stratagem enabling it to resist the current antibiotics and this would largely bypass the stratagem being developed in this proposal.”
2. In just the following paragraph:
“APPROACH, Strengths
This approach would be antibiotic independent.”
Go ahead, defend this product of NIH peer review. IMO it has NOTHING to do with my “learning how to write a proposal”.
LikeLike
August 16, 2011 at 3:59 pm
When ALL applications are discussed (e.g. at NSF) the panel develops a common stance, and the PI knows what needs to be improved/changed/addressed.
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHA!!!!!!!!!!!!!!!!!!!!!!!!
LikeLike
August 16, 2011 at 4:45 pm
“HAHAHAHAHAHA”
Fine. In 50% of the cases. At NIH, this number for triaged applications is ZERO. NULL.
Anyway, personally, I have never received any inconsistent/nonsensical critique from NSF. As for NIH, lo and behold, today I received my VERY FIRST Summary Statement that is free of bullshit of any kind. Not funded, based on: 70% priorities and perceived risks, 30% on scientific differences and probably (between the lines) that it’s the first round. I can live with this.
LikeLike
August 20, 2011 at 9:05 pm
“I have never received any inconsistent/nonsensical critique from NSF.”
And anecdote is not the plural of data. I have received such a critique from NSF.
Peer review is a process that involves human beings and is therefore inherently imperfect. Have a beer, get over yourself and write another fucking grant.
LikeLike