Your Grant in Review: Idiotic Reviewer Comments
November 5, 2010
Since the PP has been hitting it out of the park with hilarious videos and slightly more serious dissections of irrational responses to disappointing grant reviews, I’ve been thinking on one of the triggers.
Idiotic Comments.
Nothing makes a grant applicant fixate on the deficiencies of NIH peer review like an idiotic comment. We’ve all received them. The glaring mischaracterization of the literature. The utter misunderstanding of the preliminary data presented…or of the methodologies proposed. The occasional ad hominem that evades the SRO’s editorial hand. A puzzling failure to internalize the essential points made repeatedly in bullet point or bold face type throughout the application. Etc*.
But here’s the thing.
Idiotic comments have, in many cases, very little impact on the disposition of your grant. If your application is discussed at the table, then it is very likely that inaccurate and idiotic and even personally biased comments will be revealed for what they are. As I’ve said before, if your grant is to have a chance you are going to have to have won over at least one advocate. Perhaps several. One of these advocates is going to rebut misplaced comments during the course of discussion. They are going to evaluate the stupid comments as politely as possible…while still making it obvious what a screwed up criticism it really is.
The detractors often dig their own grave as well. If someone is riding a personal hobby-horse…this becomes really, really obvious in the course of discussion. They may even have a reputation (if they are a permanent member of the study section) for being sort of…blind….on a given topic and the rest of the panel discounts accordingly. [The slight caveat to this is that hobby horse riding does have the potential that a slightly cranky viewpoint** can be argued into the heads of other people over time. This, btw, is another reason to serve a full term on a study section, so that you can make your pet ideas have a little broader reach.] This is not an all-or-nothing thing, I should emphasize. In my experience, one may think a fellow member of the panel is absolutely nutz about one particular issue and still find them to be generally an excellent reviewer.
So calm down with the “flawed review” ranting. In most cases that comment that has you so exercised did not really make the difference between funding and “we advise you to revise and resubmit“.
__
*and by all means list your favorite idiotic reviewer statements in the comments
**and let us be clear, we all have our little hobby horses and biases
DrugMonkey 
November 5, 2010 at 7:44 am
One advantage of being a study section member is to promote your own hobby horse. One problem with the current system is that these biases of the study sections drive what research is done, reducing the diversity of thought given to a problem. Study sections in the days of resubmission, could micromanage many aspects of the science. At my institution, if you revised your grant to fit the critique, then your protocol had to match. No more saying something to appease crazy reviewer #3 but then doing something else (wink, wink)- you will damn well do exactly what you wrote or the ironically-named PAL will be on your ass.
As pay lines tighten even more, I’m afraid these biases will become even more critical. Any negative comment has the potential to drive a project out of the fundable range.
You’re right- one reviewer who misses some critical point and leaves a whacky comment in the critique will not torpedo most projects. I’m more worried about the attitude of “this doesn’t fit with my view of this disease or phenomena so I’m going to say no” that really thwarts novel ideas.
LikeLike
November 6, 2010 at 5:21 am
“Idiotic comments have, in many cases, very little impact on the disposition of your grant. ”
I beg to differ. In this climate, any reason to sink a grant is a reason to sink a grant, no matter how stupid. This seems to be especially true for first-time newbies. In my case, the study section dismissed an entire field (ie, bunnies don’t really hop), despite support of 6,000 publications and three nobel prizes.
I know this is anecdotal, but one fairly new member of a study section told me that despite his championing what he thought was an outstanding application, other members trounced it in favor of an older NIH club member, well funded, but who submitted a lousy application. Because of the club member’s status, errors and shortcomings within the proposal could easily be overlooked.
I guess the bottom line is that who you know IS important.
LikeLike
November 6, 2010 at 11:15 am
one fairly new member of a study section told me that despite his championing what he thought was an outstanding application, other members trounced it in favor of an older NIH club member, well funded, but who submitted a lousy application. Because of the club member’s status, errors and shortcomings within the proposal could easily be overlooked.
This appears to be arguing in line with DM’s comments, i.e., that any idiotic comments directed at your application were irrelevant and these other factors were more important…
LikeLike
November 6, 2010 at 11:27 am
You know, I’ve heard this story a million fucken times–awesome groundbreaking grant from a nobody gets trashed in favor of some piece of shitte from a “club member”–but I have never seen anything even remotely like this happen at study section.
LikeLike
November 6, 2010 at 3:15 pm
Having spent several $tens of thousands with dozens of collaborators unsuccessfully going for NSF grants for my little company, I’d love to see these “hilarious videos,” can you elaborate? Part of what we do is video, and the sting can be very well relieved with satire 🙂
Seriously, if it matters in this discussion, I and many colleagues have had horrible experiences with NSF; the directorate we submitted to clearly was dominated by the “program officer,” who ran the panel, coached the reviewers, and the idiotic comments came from him (and one year her) and sank us. Three years in a row.
I know nothing about NIH (sorry if this is wildly inappropriate), but seems to me a pre-proposal process is the thing to do, so biases and bs can be sorted out before we spend hundred of hours on a full blown proposal.
LikeLike
November 7, 2010 at 4:50 am
NIH does have a quasi/unofficial pre-review. Its called “talk to your program officer”. And for some mechanisms, such as K-awards, they want to see a draft of the Specific Aims before submission to avoid such problems.
I don’t think a formal pre-review process would work. I’ve been triaged (and if you haven’t, you’re not submitting enough grants or enough interesting grants), and the reviews I got were for the most part very useful. A lesser/pre review would have been less useful. From a study section’s standpoint, adding a level of review smells of more work, or worse, avoiding the 2-submission limit.
Finally, I second what CPP says above. I’ve never seen anything so egregious happen at study section. People who are rejected say its not fair, people who are funded say it was. But if you ask the people who are not invested in the outcome of any particular grant (beyond helping the good ones), the consensus is that its largely fair, and that its just hard to get funded.
LikeLike
November 7, 2010 at 8:36 am
People who are rejected say its not fair, people who are funded say it was.
By this point I am reasonably consistent in my reaction to the disposition of my applications, whether they are in the fundable range or not. When it comes to “fair”, it is just as inequitable, so to speak, if I should land a 5% as if I should land a 25%. I assume that sometimes the random variation goes my way and sometimes it does not but overall, my grants are all in the “pretty decent” category. I really would defy anyone to show in objective terms where the top scoring few of my applications are “better” than the ones that came into the just-missed category of scores.
I actually think it is a pretty serious mistake to obsess over fair/unfair and whether or not a particular round of review was “correct” or not. This sort of thinking gets in the way of doing your job which is to land grant funding, by means fair or otherwise, and get on with doing some interesting science.
LikeLike
November 7, 2010 at 8:39 am
I’d love to see these “hilarious videos,” can you elaborate?
whoops, I managed to forget the link when I originally wrote this. I meant to link to
This post at Comrade PhysioProf and the following video:
LikeLike
November 7, 2010 at 2:13 pm
HAHAHAHAH! I really cracke myself the fucke uppe!
LikeLike
November 7, 2010 at 8:45 pm
PIs who have finally received their first triage in their 35 years of National Academy of Science inducting work and are OuTRaGEd!!11!! crack me the fuck up.
LikeLike
November 8, 2010 at 5:26 am
I haven’t seen “awesome groundbreaking grant from a nobody gets trashed in favor of some piece of shitte from a “club member”, but I have seen the benefit of the doubt given to established researchers but not others. For example, some labs are “trusted” to do things right and can submit proposals without so much methodological detail, but newbies cannot.
LikeLike
November 8, 2010 at 5:43 am
I concur with anonym. There is a slant toward established investigator’s proposals in review.
LikeLike
October 29, 2011 at 6:17 am
This really cracks me up. “I haven’t seen “awesome groundbreaking grant from a nobody gets trashed in favor of some piece of shitte from a “club member”
I wish this were not true. On the very last study section I sat on, I pointed out that a whole aim a very well established PI with Science and Nature papers under his belt (nothing against these people, I am one too) was proposing was literally impossible, because he had not understood something basic about a new technique. For that we would sink a new investigator with the appropriate ‘go back and do your homework son’ comments. But my pointing this out was met with a kind of stunned silence by the other reviewers who had been praising this man’s record as a main reason for scoring him high. Others nervously joined in: “are you sure, how do you know its impossible…” the primary reviewers began to sputter objections and discount this. The section began to chat about this and people and people wondered if the applicant knew something I didnt…. Then, another usually very quiet member who was not a designated reviewer spoke up and said: “Marcus is right, completely. The aim is impossible.” And spelled out in exquisite detail why [ she was using the technique and knew its limitations cold]. Even then one of the primary reviewers stubbornly held on to his “1” recommendation.
My point here is not to tell my heroic study section story, but that I truly believe new investigators are held to entirely different standards on writing specific clear feasible aims. I know there are counterbalancing problems established PIs face.
But lets face it, though I sense that DM is very positive about the present system and his basic advice seems to be “write more grants and saturate the system till you win the lottery”, there are so many problems with the way peer review works and lazy smug reviews are the most maddening experience anyone has to deal with in scientific life because THERE IS NOTHING YOU CAN DO ABOUT IT. YOU JUST HAVE TO TAKE IT. IF YOU POINT OUT THE REVIEWER IS STUPID AND BIASED OR CLUELESS — OR EVEN JUST MISREAD ACTUAL WORDS TO GIVE YOUR PROPOSED EXPERIMENT A CRAZY ANGLE — IN THE RESPONSE YOU GET A SMIRKING “unfortunately the applicant is not responsive to the critcisms but chooses to merely argue…” or you get an indignant LOWERING of the score by the now ego-challenged reviewer.
LikeLike
October 29, 2011 at 7:21 am
You are apparently a newcomer to this blog and it is true that lately I’ve found myself criticizing people who expect their unique-flower status makes their treatment under the system particularly notable. I have my problems with the system and have talked about the dismal treatment of junior investigators repeatedly in the past.
In terms of solutions, what I urge is precisely what you describe doing- speaking up as a study section member. As you note, sometimes a comment from a loudmouth empowers others to join in the discussion. Also, I try to advocate for a real, workable (albeit small) solution to this problem- junior scientist representation on panels.
LikeLike
October 29, 2011 at 7:26 am
I usually recommend people start with this post
if they want to know where I am coming from, blogwise.
LikeLike
October 29, 2011 at 7:38 am
I have also seen the opposite outcome of Marcus1171, where a study section piles on a senior investigator for making the kinds of grantsmanship mistakes that n00bs can get the benefit of the doubt over.
LikeLike
October 29, 2011 at 8:09 am
Fine. But It is not clear to me that having young investigators on the panel makes that much difference. In my experience, they are often nervous, feel the need to prove they are equal to the task and how tough they can be and come in with long lists of complaints about proposals, and are not able or willing to take a broad view or stand up for a risky idea. They are afraid if they assign a 1 or a 2 they may be viewed as soft.
The best reviewers are often very senior people who can be above the fray, don’t give a crap about any politics, will take the long view and call like they see them based on a serious line by line reading and study of a proposal. Unfortunately such wisdom and confidence born of success, combined with a generous and fair temperament….. is rare!
LikeLike
October 29, 2011 at 10:06 am
Don’t keep us in suspense Marcus… what score did you give this grant?
LikeLike
October 29, 2011 at 3:33 pm
Marcus,
Most of your objections, if even true*, are attributable in large part to reviewing experience- simple fix there.
The “best” are hardly ever the oldsters, too steeped in all kinds of biases against the young, fanboi of their oldster buds, ignorant of new approaches, etc
*not so in my experience.
LikeLike