Peer Review: Friends and Enemies

Perhaps “advocates” and “detractors” are the better terms. This is one of those heuristics that might help with crafting responses to the Summary Statement or the paper review. Others have views that touch on the topic for example MWE&G has the following in a recent post:

if you’ve hooked your primary reviewer into being a passionate advocate for your proposal, that will likely come through as well. If the summary statement lacks any sign of someone going to bat for your work, then you did not make your case even to the reviewer who should have been most excited about your project.

The heuristic is this. In situations of scientific evaluation, whether this be manuscript peer-review, grant application review, job application or the tenure decision, one is going to have a set of advocates in favor of one’s case and detractors who are against. The usual caveats apply to such a strict polarization. Sometimes you will have no advocates, in which case you are sunk anyway so that case isn’t worth discussing. The same reviewer can simultaneously express pro and con views but as we’ll discuss this is just a special case. Etc. Nevertheless there are a couple of points which arise from this heuristic which apply to all of the above situations and suggest concrete approaches to both original presentation and, where applicable, in revising the proposal/manuscript. We’ll take the case in which one is crafting a revision to a grant in response to a prior critique as the example after the jump.

Give your advocates what they need to go to bat for you.

This is the biggie. In all things you have to give the advocate something to work with. It does not have to be overwhelming evidence, just something. Let’s face it, how many times are you really in position in science to overwhelm objections with the stupendous power of your argument and data to the point where the most confirmed critic cries “Uncle”. Right. Never happens. So why write a response that seems to feel you can do just this with “rebuttal” language?

Let’s take some stock critiques as examples. “Productivity”. The goal here is not to somehow rush 8 first author papers into press. Not at all. Just give them one or two more papers, that’s enough. Sometimes reiterating the difficulty of the model or the longitudinal nature of the study might be enough. “Independence of untried PI with NonTenureTrackSoundin’ title”. Yes, you are still in the BigPIs lab, nothing to be done about that. But emphasize your role in supervising whole projects, running aspects of the program, etc. It doesn’t have to be meticulously documented, just state it and show some sort of evidence. Like your string of first and second authorships on the papers from that part of the program. “Not hypothesis driven”. Sure, well sometimes we propose methodological experiments, sometimes the outcome is truly a matter of empirical description and sometimes the results will be useful no matter how it comes out so why bother with some bogus bet on a hypothesis? Because if you state one, this stock critique is de-fanged, it is much easier to argue the merits of a given hypothesis than it is the merits of the lack of a hypothesis. “Applicant really needs to add the following control conditions and these extra manipulations…” Yes, yes, the reviewer is out to lunch and you can dispose of these silly observations with a few pithy rejoinders. Don’t do this. It puts your advocate in the hole. Just give them something, after all, you don’t have to actually do everything you propose!

Deny your detractors grist for their mill.

Should be simple, but isn’t. Particularly when the critique is basically a reviewer trying to tell you to conduct the science the way s/he would if they were the PI. (An all to common and inappropriate approach in my view) If someone wants you to cut something minor out, for no apparent reason (like say the marginal cost of doing that particular experiment is low), just do it. Add that extra control condition. Respond to all of their critiques with something, even if it is not exactly what the reviewer is suggesting; again your ultimate audience is the advocate, not the detractor. Don’t ignore anything major. This way, they can’t say you “didn’t respond to critique”. They may not like the quality of the response you provide, but arguing about this is tougher in the face of your advocating reviewer.

Give the critical reviewer the excuse s/he needs to shift to advocacy.

Perhaps more of a subcategory of the above two points but worth addressing for one reason. It happens. Frequently. A reviewer that may sound highly critical on the first round may switch to favorable on the next revision; even without the sort of fundamental change that s/he is apparently envisioning. There are many reasons for this and it relates to the overall high quality of grants well down into the 60th percentile and the inevitable competition between grants stacked up in the given reviewer’s pile of applications. Not to mention a huge bias for revision (“I’d like to see this one come back”). The point for the current discussion is that number one you don’t want to insult the critical reviewer too badly and two, this is why you need to address all concerns as best you can. This way even the critical reviewer can backtrack with little loss of face “Well, the applicant didn’t fully satisfy the critique but they did x,y and z which I have to admit is an ideal compromise under the constraints”. This happens. Frequently. Otherwise how would the vast majority of grants be funding after revision?

Plan ahead

Although the examples I use are from the grant review process, the application to paper review and job hunts are obvious with a little thought. This brings me to the use of this heuristic in advance to shape your choices. Postdocs, for example, often feel they don’t have to think about grant writing because they aren’t allowed to at present, may never get that job and if they do they can deal with it later. This is an error. The advocate/detractor heuristic suggests that postdocs make choices to expend some effort in broad range of areas. It suggests that it is a bad idea to gamble on the BIG PAPER approach if this means that you are not going to publish anything else. An advocate on a job search committee can work much more easily with the dearth of Science papers than s/he can a dearth of any pubs whatsoever! The heuristic suggests that going to the effort of teaching just one or two courses can pay off- you never know if you’ll be seeking a primarily-teaching job after all. Nor when “some evidence of teaching ability” will be the difference between you and the next applicant for a job. Take on that series of time-depleting undergraduate interns in the lab so that you can later describe your supervisory roles in the laboratory.