Your Grant in Review: Preempting conflicted reviewers
February 24, 2011
Well, score this one in the file of YHN learns something new.
One of the fondest accusations and complaints of unsuccessful NIH grant applicants is that their application was reviewed by someone who is in scientific conflict with their proposal. Therefore the unsuccessful outcome was related more to selfish interest on the part of one or more reviewers than it was to the merits of the proposal itself.
This accusation generally boils down to one of two things
1) The other reviewer represents the other side of a scientific debate and is motivated to squelch your application so as to continue to “win” the scientific debate by default, rather than in the competition of equals who are both (or all) funded sufficiently to carry on the debate with data, logic and papers.
2) The other reviewer is on the same side of a scientific question but is motivated to squelch your application so as to be able to be the one to publish the anticipated findings.
There are, of course, refinements on these themes. Perhaps it is that a buddy is the person with the “interest”. Perhaps it is thinly veiled payback for perceived prior slights by the applicant or the applicant’s training pedigree. Whatever the details, the point is that the applicant feels as though her chances would have been much better with another reviewer.
I’ve written a fair bit about how pointless it is to appeal the review after the fact. My suspicion is that since many, many unsuccessful applicants cry about reviewer conflict on the thinnest of evidence, the NIH is motivated to make the review process seem as futile as they possibly can.
Occasionally, however, I get a question from a newer applicant regarding how to preempt the conflicted reviewer. Perhaps it is a revision application and the reviewer in question is likely to get the application again. Perhaps the person is a standing member of the study section that is desired.
My standard advice has been to try to find another study section, under the impression that complaints about reviewer conflict don’t hold much water with SROs. Again, it is my suspicion that so many PIs would try to claim this that the entire process would break down. I mean seriously, how do you come up with 30 people that are expert on a study section sized theme of science, some quarter of which probably are in a given subfield to which an application belongs, and not have the specter of scientific conflict arise? It seems inevitable to me.
So my version of the cover letter is short and sweet. “I ask that this be reviewed by GRZLLP study section because [insert buzzwords that overlap between your application and the CSR official description of the study section]. I also request that it be assigned to [insert your favorite IC] because of blumbelty mumble obvious reasons”. That’s it. Short and to the point.
Consequently I listened to a recent podcast published by the Office of Extramural Research with some surprise. In the middle of the one on Cover Letters (dated 2/18/2011) a recommendation is given by Dr Ann Clark, Associate Director of CSR’s Division of Receipt and Referral, to list conflicted reviewers who you wish not to review your grant right in your cover letter. With the reason for the conflict.
Honestly, I’m flabbergasted by this official recommendation.
Live and learn, live and learn
DrugMonkey 
February 24, 2011 at 2:19 pm
Here is my situation. What would you suggest I do, besides talk to the program officer?
Reviewer 1: impact 1.5
Reviewer 2: impact 2.5
Reviewer 3: impact 6.5, with a detailed, 3-page aggressively critical soliloquy of commentary. A few of the comments are relevant, but a majority are not. The reviewer claimed that I had a poor understanding of the current literature and that the proposed experiments were too difficult. Basically, it went beyond the realm of criticism and into the realm of hate.
I can make an educated guess about who the reviewer is, but of course I can’t be sure. When I resubmit, do I ask for this person to be excluded? What if I have guessed wrong? There is not an obvious conflict of interest in terms of the research area. It is more of a conflict of opinions about the controversial issues in the field.
LikeLike
February 24, 2011 at 2:55 pm
I listed right on the cover letter several pi’s that are in conflict for my recent r01, which btw beaker, was similarly scored- w respect to rev #3 (ie Disscussant) pull down. I believe conflictive pi’s I listed were indeed excluded, despite rev 3. I think I’m gonna address the critique head on w solid arguments and data, but do it NICE way!
LikeLike
February 24, 2011 at 3:50 pm
Hmmm … I thought listing potential reviewer conflicts in the cover letter was standard practice. I was told to do this with all of my NIH submissions as Postdoc Mentor is on the study section. That’s a pretty straightforward conflict, though … I’m guessing that saying “Potential Reviewer hates me and won’t say nice things about my grant” isn’t a good thing to do.
Oh, it’s also clearly stated here: http://cms.csr.nih.gov/ResourcesforApplicants/Submission+And+Assignment+Process.htm
LikeLike
February 24, 2011 at 4:33 pm
It’s unlikely I will ever apply for NIH funding, but I love these posts as I get to learn about the many, many differences that exist between different fields and funding agencies. For instance, one DOE program manager once told me explicitly that writing whom not to send the proposal to (there is a spot to provide that information in the submission form) is very bad form. If you absolutely must communicate this information, do it only over the phone to the program manager.
LikeLike
February 24, 2011 at 6:38 pm
Re: Saban_Lab. Yea, the bottom line is that I must address the negative reviewer with solid arguments and data. There is no other way forward. The first draft of the reply will not be nice, but the final version will be.
LikeLike
February 25, 2011 at 4:59 am
It is impossible that (1) individual impact scores can be anything other than whole numbers and (2) that you can know what impact scores were assigned by any particular reviewers. Where are you coming up with this bullshitte?
This is horrible practice for maximizing the likelihood of your application being assigned to the institute and study section you desire. You need to talk/e-mail to a program officer and scientific review officer and gain their assent that your application should be assigned to their institute and study section, respectively, before you submit, and you need to refer to those people by name in your cover letter as having agreed that those assignments should be made.
I have *never* had an assignment made counter to my desires applying this strategy, and I have been told by people who know that Division of Receipt and Referral always goes ahead and assigns applications to such institutes and study sections if POs and SROs are referred to by name. From what I’m told, they figure they may as well just save time looking at the application and go along with the cover letter in such a case, and let the PO and/or SRO bounce it back if the assignment is wrong or the applicant is full of shit.
To the contrary, I am aware of numerous cases where applicant requests for assignment of institute and/or study section are ignored where the applicant nakedly asserts that particular assignments in her own opinion are suitable.
LikeLike
February 25, 2011 at 5:16 am
Clarification. CPP is correct. The truth is that the third reviewer’s preliminary scores were so low that that grant was not discussed. The “imapct” scores I wroe are just an average of the various criteria. I know those don’t necessarily reflect the overall score, but that’s all I have to go by.
LikeLike
February 25, 2011 at 6:32 am
Yes…as a longtime Scientific Review Officer, I can assure you that we DO eliminate anyone the applicant lists as a conflict, as long as a credible reason is provided. However, a word of warning: NEVER, EVER, EVER list the names of people you might “suggest” to review it. While that may be acceptable for some journal papers, it will guarantee that those people WILL NOT be asked to review your application, even if prior to seeing your request, the SRO had planned to give it to them.
LikeLike
February 25, 2011 at 7:32 am
PP, I’ve never had a problem with getting it *originally* routed to the place I wanted it to go. A time or two I’ve had one re-routed and once been truly screwed because of this. But, it is dollars to donuts if I’d done the preliminary dance the SRO would have told me it had to go to that other section anyway. Not sure where getting this redirection in advance is any different than it happening after the app has been submitted.
ETA: I should note that once I’ve been pretty well hosed because of not using the opportunity to whinge about scientific conflict until late in the game. I got the A2 jacked out of the bad study section but by then it was a lost cause…
LikeLike
February 25, 2011 at 7:38 am
Beaker, the other thing you have to go on are the criterion score-overall impact score correlation data provided by NIGMS showing that Approach and Significance still carry the day. So if you are looking at individual criterion scores, you should weight Approach and Significance more heavily in trying to conclude what the impact score was.
your averages of 1.5 and 2.5 suggest that two of the reviewers wanted the application discussed. So it should have been discussed. One reviewer by themselves might have been willing to shut up but when it is 2/3 like this? Seems unlikely
The fact that it was not discussed suggests that Rev #3 was either highly convincing to those other two, or that they each had a driver (perhaps the Approach or Significance) criterion score that was more a 4-5 range
LikeLike
February 25, 2011 at 8:25 am
You never truly know who your friends and enemies are.
While not exactly the same, I suppose the approach should be similar to asking for specific people to be excluded from reviewing your manuscript. I would say though that while you may think a particular person doesn’t like you, your work, or can’t objectively review your paper, you may well be wrong.
I had a ms where we asked a particular person not to serve as a reviewer. He did review it, and it was good he did. It was the most in depth and thoughtful review I have received. He showed a shockingly complete understanding of all aspects of our lab’s work and made very specific and doable experimental requests. He went on to comment to my PI in person how much he liked the paper and even wrote a favorable Faculty 1000 review.
LikeLike
February 25, 2011 at 8:26 am
DM. Yup. If you take the Approach score only, I got 2, 3 and 8. And yes, reviewer 3 knew the field very well. S(he) pulled no punches and beat my proposal to a pulp.
LikeLike
February 25, 2011 at 8:38 am
Its a tough world. I saw a 2, 3 and 5 triaged in our study section. It all depends on the other scores.
LikeLike
February 25, 2011 at 9:03 am
Re: Beaker, 2/24 2:19 pm
For one 01 application a couple of years ago I had a similar scenario, though it was under the old scoring system. I had 4 reviewers, 2 liked it, 1 was moderately critical but the criticisms were legit, and 1 was clearly a hater with a conflict of opinion about a somewhat controversial issue in the field. It was not triaged but scored poorly. When I submitted my A1 revision, I called and talked to the SRO about my concerns about the hater, although I did not know who this person was. She agreed to list “reviewer 4” as a conflict of interest and not to assign my grant to this person. Not that it got me funded, so it’s hard to say if it was really helpful to do this. Still, it seems like if you have a concern about a biased reviewer whose biases are reflected in their review comments, at least with some SROs, it should be possible to exclude them on conflict grounds without knowing their identities.
LikeLike
February 26, 2011 at 4:01 am
The problem these days is that, in many study sections, only 40% of applications from established investigators can be reviewed.
With all awardees from the stimulus period coming up for the renewal of their 3-year R01s, SROs are receiving >110 applications per cycle.
A negative review is all you need to get your application in the “unscored” zone. Now, what I have seen is that all applications that were not scored had at least a moderate weakness. Reviewers, in general, read the comments of other reviewers when they strongly disagree with their view before the actual meeting (I do, at least).
Competition is fierce and, with the new scoring system, everything in your grant needs to be just impeccable because one point in your impact score is the difference between life and death.
It is hard for everybody.
LikeLike
February 26, 2011 at 4:26 am
The impact score cutoff for triage obviously depends very heavily on the typical scoring behavior of that study section. One study section I have served on is typically very inflated, and the triage cutoff is usually in the low threes. This cutoff would be at a level of “only minor weaknesses”.
On another study section I have served on scores are much less inflated, and the triage cutoff is usually around five, which involves a moderate weakness.
LikeLike
February 26, 2011 at 8:57 am
However, a word of warning: NEVER, EVER, EVER list the names of people you might “suggest” to review it.
This advice is specific to the NIH only. The NSF for example has the exact opposite policy, where you are *encouraged* to submit a list of suggested reviewers (as well as reviewers to be excluded).
LikeLike
February 26, 2011 at 12:42 pm
“However, a word of warning: NEVER, EVER, EVER list the names of people you might “suggest” to review it.”
I have a different experience. I was once asked by an SRA to give a list of reviewers ~10 with expertise in my area. I did but it is also true that I never got any of those reviewers to review my application.
LikeLike
February 26, 2011 at 3:35 pm
The advice not to put it in a cover letter is consistent. But one to one contact with SRO is another matter. I’ve mentioned specific ppl and had them appear on the roster. Could be coincidence though…
LikeLike
October 27, 2011 at 10:38 am
But here’s what happened to me. What to do? I am a senior PI, too got 2,6,3 scores for approach on a renewal this fall and got triaged for the very first time in my career. I then looked at the study section roster again and and noticed that the PI of a lab I am collaborating with was on it. I noticed this the week before the review took place too but had not thought much of it, thinking he would recuse as he is collaborating this very year with me. I had not thought to exclude him when I sent in the grant app as I did not see him as prominent in the field and he was certainly not on any prior rosters (in other words this was his first time on the section).
In the collaboration, this guy was very demanding, wanted us to do all sorts of things we could not, send him biologicals that take us weeks and personnel time to prepare, etc. He was disgruntled. I am pretty sure he was reviewer 2 but of course cannot know for sure unless I trigger an investigation.
Reading critique 2, I think it is him. I called the program officer and he is very dismissive, says he knows of no way to investigate that. Should I do a formal appeal? If I just resubmit, I am worried that having had it triage and with the new reviewers seeing all his negative comments, I wont go over the 10% payline.
LikeLike
October 27, 2011 at 11:24 am
Marcus,
I would think that this reviewer would have alerted the SRO that he/she is collaborating with you and, at minimum, would have asked if the situation represented a potential conflict of interest, don’t you think?.
LikeLike
October 27, 2011 at 11:26 am
I am a senior PI,… got triaged for the very first time in my career.
I would like you to listen VERY carefully to me. Experience says this will be futile but, what the heck. Let’s give it a go.
Welcome to reality street. Welcome to exactly the same thing the people more junior than you have been up against since the start of their careers.
Yes. Including the disparate scores, the possibility of being sunk by a biased reviewer and the shock of having your amazing science dismissed by someone you don’t think of as “prominent in the field”. Including the seemingly deaf ear of the PO to your unique (!) set of woes and your highly meritorious science.
Your case is no different than anyone else. You need to get this. At a very deep, hindbrain level without a covering “yeah but” defense mechanism. This is the gig. It always has been the gig for many of your younger colleagues. It is just that now, finally, the funding is so tight that it is affecting you too.
I would advise you to take the same approach that many of us have adopted. To knuckle down, accept that this is the current reality and to respond accordingly. My best strategic suggestion is that you need to write more grants, of course. Find multiple ways to bite the apple. Search out a few different study sections. Consider several different grant mechanism, maybe even organize that Program Project you’ve been thinking about.
And above all else, never, ever, expect that a competing continuation will just sail through as it used to.
LikeLike
October 27, 2011 at 11:34 am
I would think that this reviewer would have alerted the SRO that he/she is collaborating with you and, at minimum, would have asked if the situation represented a potential conflict of interest, don’t you think?
This is supposed to happen, yes. Current collaboration is grounds for a COI and for that reviewer to be recused from your application. This makes it less likely that Marcus is correct about the identity of Reviewer #2 for his/her application.
Of course, I believe the expectation is that your collaborators will be too kind to your application rather than the reverse, no?
LikeLike
October 27, 2011 at 11:44 am
Yes. It is assumed that virtually all collaborations are to collaborate not to “miscompete”.
LikeLike
October 27, 2011 at 11:46 am
I probably did not make this clear. The guy was disgruntled, wanted us to to do things I told him we couldnt. Basically wanted us to be a service to make complex products for him. I told him we couldnt do it. So the collaboration fizzled in March with this guy not happy. The grant was submitted this summer for review in October.
LikeLike
October 27, 2011 at 11:59 am
The more disgruntled, the less likely he/she wanted to review a grant of a failed collaboration. In my opinion, it is too risky and not very smart for someone to proceed in the way you think this reviewer has behaved.
LikeLike
October 27, 2011 at 12:45 pm
I understand your specifics Marcus. I was referring to the reason why collaborative relationships are one of the default, presumed COIs. Like mentoring relationship, co-employment and marriage.
LikeLike
October 27, 2011 at 12:50 pm
That would indeed be my approach in such a situation.
But I have reason to believe from some comments in the critique that he did it anyway. If so, it seems to me this is a clear cut case of reviewer malfeasance and that gives me grounds for succeeding on a formal appeal and having it re-reviewed by a special emphasis panel rather than this panel. The only question is — is that worth it? Council wont meet until January.
But what have I got to lose? Have you ever seen a grant make it from triage to top 10% in revision. I cant recall one in the last 5 years that I have sat on [a different] study section.
I realize that drugmonkey thinks that I just dont get it and this is the new “reality”. but even if they payline was 25% I would be in the same boat. I also realize it may seem like I’m delusionally blaming the reviewer (“Its always reviewer 3! like in the Downfall movie spoof). But this business is hard enough without this kind of twisted review (i am sparing you a recounting of how transparently “assassin” the review was).
LikeLike
October 27, 2011 at 2:05 pm
If you have nothing else better to do with your time, sure, appeal away. Who knows? Maybe you are right and it *was* the collaborator. Maybe you are one of the very few that will succeed in winning your appeal, have it sent back to review and manage to get a fundable score. You could be the one.
But absent a great deal more information, we have to assume this situation is just like the rest. In fact your “first triage ever” comment sets you in a particular category that is familiar to me. A category of investigator which tends to assume their case for being outraged is stronger than it is. Because they have not been through the school of hard knocks on grant review.
Others have. Yes, even to the extent of weathering “transparently assassin” and “twisted” critiques. Unjustifiable outcomes. Losing grant funds that they probably deserved, in some omnipotent sense.
My only question for you would be, how sympathetic have you been in the past when junior folks complained about similar problems with review? About bias in the system against the young and the new? How do you react to accusations of zip code bias? What did you think in your heart of hearts when your colleagues reported being triaged?
Did you think, overtly or covertly, that you were just a better scientist and that those people somehow deserved their fate? Or did you feel the same sort of outrage about the limitations of the peer review system that you are expressing now?
Really….I’m curious about this one.
LikeLike
October 27, 2011 at 2:48 pm
Marcus,
The guy in the lab next to mine just got an 8% after having been triaged in the prior review. I’m sure there are many more examples now that A2’s have gone away.
LikeLike
October 27, 2011 at 3:39 pm
If you were triaged, it means neither of the other two reviewers thought it was worth spending time discussing the proposal either. You’re not going to win this one no matter what the outcome of an appeal would be.
LikeLike