All Peer Reviewers are Equal, But Some Peer Reviewers are More Equal Than Others
March 4, 2008
The peer-review of scientific papers is in one sense democratic and in another sense highly authoritarian and dictatorial. What is most important is that the scientific peers with the most appropriate level of expertise review a given manuscript which is seemingly democratic. What is most critical, after all, is that the science itself be reviewed with the greatest scrutiny and held to the highest standard, right? The identity, status and formal credentials of the reviewers are less important than is the specific type of expertise.
So what is up with Editorial Boards?
Many scientific journals maintain an Editorial Board of scientists who are usually on the more-senior side of things and represent the breadth of science that is typically published in the journal. Check the website of your favorite journal or three and there should be a link for “About this journal” or some such. The Editorial Board member list may be a few clicks away but you should find it somewhere. Recognize the names? Of course you do because they are, at the least, recognized members of one or more of the fields represented in the journal. Most often, a fairly active and well-respected member to boot. Sometimes you will get a distinct impression of “representation” of more than just scientific subfields. Gender and geographic representation in some cases. University vs. Research Institute vs. Industry in some other cases.
Ok, so the journal has these lists of scientists on the Editorial Board. What are they doing? Naturally one reason for these Boards is to enhance the reputation of the journal itself. Especially lower-down on the totem pole a journal might enhance its respectability by the quality of people willing to serve on the Editorial Board, no?
The real role of the EB members is, however, to function as the “more equal” reviewers. The members are there to handle a more-than-random amount of reviewing for the journal. I don’t have data on the variety of journal practices although I’m sure they vary. It would be hard to tie down how many “extra” reviews one would be asked to write just by being on the Board but suffice it to say one is committing to some extra work. In most cases where I’ve heard details, this situation is made pretty explicit in advance. So automatically this EB reviewer is going to have an increased impact on the peer-review practices of that journal in his/her area simply by doing more reviews than average.
Next we get to the degree of influence a given reviewer has over the disposition of the manuscript. It should be no secret to most that one of the jobs of the Editor is to choose sides amongst the reviews received for a given paper. Sure, many times the reviews communicate a similar message. But in a whole bunch of cases it is clear that the reviews disagree to larger or smaller extent. As someone who has received a fair number of mixed reviews and been a reviewer where another reviewer didn’t agree with me, it is clear to me that the Editors have done a lot of executive decision making with respect to which reviewer to “side” with. What influences the Editor’s decision? Well, clearly the quality of the review plays a big role. If you write clearly, dispassionately and support your points well it is more likely that the Editor is going to side with you. If you toss off some one-liner opinions without clear links to the manuscript, well…. It strikes me, however, that this is another area in which the EB reviewer may have more-than-equal influence given an equal quality of the review itself.
Moving along, the EB reviewer may also function as a tie-breaker in a more explicit sense. I’ve run across more than one situation in which I’ve been asked, in essence, to tie-break or review the reviews. As in the Editor handling a manuscript asked for a review of a specific point of contention between reviewers (or between reviewer and author on revision). Or I was asked for a “quick” review of the manuscript with the notation that the Editor was holding conflicting reviews and they needed a good third review to make a decision. Etc. Obviously this does not require that an Editor resort to the EB members. This is, however, one additional and fairly explicit role fulfilled by Editorial Boards and is therefore yet another way in which the influence of EB members is more-equal.
Is there anything wrong here DearReader? Does the EB fulfill a critical role in the publication process? Is it a key part of getting high-quality reviews? A key part of keeping the journal on-task with respect to quality, audience and mission?
Or is it an inherently conservative feature? A perpetuation of GoodOldBoyzNGirlz clubs? Do EBs prioritize navel-gazing, as in “this is the way we construe GoodScience in our field and never it shall change”?