All Peer Reviewers are Equal, But Some Peer Reviewers are More Equal Than Others

March 4, 2008

The peer-review of scientific papers is in one sense democratic and in another sense highly authoritarian and dictatorial. What is most important is that the scientific peers with the most appropriate level of expertise review a given manuscript which is seemingly democratic. What is most critical, after all, is that the science itself be reviewed with the greatest scrutiny and held to the highest standard, right? The identity, status and formal credentials of the reviewers are less important than is the specific type of expertise.
So what is up with Editorial Boards?

Many scientific journals maintain an Editorial Board of scientists who are usually on the more-senior side of things and represent the breadth of science that is typically published in the journal. Check the website of your favorite journal or three and there should be a link for “About this journal” or some such. The Editorial Board member list may be a few clicks away but you should find it somewhere. Recognize the names? Of course you do because they are, at the least, recognized members of one or more of the fields represented in the journal. Most often, a fairly active and well-respected member to boot. Sometimes you will get a distinct impression of “representation” of more than just scientific subfields. Gender and geographic representation in some cases. University vs. Research Institute vs. Industry in some other cases.
Ok, so the journal has these lists of scientists on the Editorial Board. What are they doing? Naturally one reason for these Boards is to enhance the reputation of the journal itself. Especially lower-down on the totem pole a journal might enhance its respectability by the quality of people willing to serve on the Editorial Board, no?
The real role of the EB members is, however, to function as the “more equal” reviewers. The members are there to handle a more-than-random amount of reviewing for the journal. I don’t have data on the variety of journal practices although I’m sure they vary. It would be hard to tie down how many “extra” reviews one would be asked to write just by being on the Board but suffice it to say one is committing to some extra work. In most cases where I’ve heard details, this situation is made pretty explicit in advance. So automatically this EB reviewer is going to have an increased impact on the peer-review practices of that journal in his/her area simply by doing more reviews than average.
Next we get to the degree of influence a given reviewer has over the disposition of the manuscript. It should be no secret to most that one of the jobs of the Editor is to choose sides amongst the reviews received for a given paper. Sure, many times the reviews communicate a similar message. But in a whole bunch of cases it is clear that the reviews disagree to larger or smaller extent. As someone who has received a fair number of mixed reviews and been a reviewer where another reviewer didn’t agree with me, it is clear to me that the Editors have done a lot of executive decision making with respect to which reviewer to “side” with. What influences the Editor’s decision? Well, clearly the quality of the review plays a big role. If you write clearly, dispassionately and support your points well it is more likely that the Editor is going to side with you. If you toss off some one-liner opinions without clear links to the manuscript, well…. It strikes me, however, that this is another area in which the EB reviewer may have more-than-equal influence given an equal quality of the review itself.
Moving along, the EB reviewer may also function as a tie-breaker in a more explicit sense. I’ve run across more than one situation in which I’ve been asked, in essence, to tie-break or review the reviews. As in the Editor handling a manuscript asked for a review of a specific point of contention between reviewers (or between reviewer and author on revision). Or I was asked for a “quick” review of the manuscript with the notation that the Editor was holding conflicting reviews and they needed a good third review to make a decision. Etc. Obviously this does not require that an Editor resort to the EB members. This is, however, one additional and fairly explicit role fulfilled by Editorial Boards and is therefore yet another way in which the influence of EB members is more-equal.
Is there anything wrong here DearReader? Does the EB fulfill a critical role in the publication process? Is it a key part of getting high-quality reviews? A key part of keeping the journal on-task with respect to quality, audience and mission?
Or is it an inherently conservative feature? A perpetuation of GoodOldBoyzNGirlz clubs? Do EBs prioritize navel-gazing, as in “this is the way we construe GoodScience in our field and never it shall change”?

11 Responses to “All Peer Reviewers are Equal, But Some Peer Reviewers are More Equal Than Others”

  1. MRW Says:

    Hmm, it seems to me that the main function of editorial boards is to set the direction of the journal, at least that’s the impression I’ve always gotten when I’ve heard about what goes on at editorial board meetings.


  2. CC Says:

    IIRC, Noah mentioned on one of your older posts that he consciously weights the reviews of experienced reviewers over those of younger reviewers. As someone who falls in the latter category, and recognizes the potential traps he mentioned, I don’t think there’s anything inappropriate about it.


  3. Noah Gray Says:

    Reviewers differ in their abilities. Basta.
    As CC mentions, I think that new reviewers follow a learning curve just as one would with anything unfamiliar. The newer (not always younger…) reviewers may learn from reading the other reviews, especially if s/he is out of step with the others. This is a practice that is typically started during graduate school when the PI allows the student to “tag along” and help him/her review a manuscript. My former advisor had me review a paper, and then we exchanged reviews. Although I did have some additional comments to add not included in his report, I learned a great deal as to what technical issues really matter, how tough to be, and how to develop an editorial instinct (some reviewers never get this and pass anything that is technically valid without providing some perspective on novelty or impact).
    A reviewer was once floored by the discrepancy between his/her review of a revision and those of his/her peers. After receiving all of the reviews from me following our decision on a particular paper (it is a common practice to send the other reviews to the other referees), this particular reviewer wrote back to me and apologized for not recognizing the shortcomings of the study. S/he then mentioned s/he planned to study the other reviews to see “where things went wrong”. Now that is what I like to hear…


  4. bsci Says:

    EB system makes sense in that senior people do have a better overview of a whole field. They should review more since there is more they should be capable of reviewing well.
    Still, it seems the problems have more to do with who is put on an editorial board and which members are assigned to with papers. A lot of this comes back to an editor. Does an editor select senior people who are known for trying to sink competitors’ work? Does the editor regularly give such a person a prominent role in reviewing the work of direct competitors? Does the editor value a 10 line rejection from an EB member more than a multi-page request for revisions that detail the flaws and strengths of the paper?


  5. Noah Gray Says:

    1. Authors always give a list of those investigators who they would rather not review their work. These lists are always honored.
    2. Reviewers are obligated to excuse themselves from the review process if they have a conflict of interest. I get this all the time. Trust me, a lot of people are dying for excuses to get out of reviewing!
    3. Editors are somewhat aware of quibbling between labs since we are so close to the action all the time. If you want to do your job right, you need to know the gossip!
    4. Weighting is done according to who the reviewer is, how appropriate they are to the crux of the matter at hand (for example, if an anatomist starts picking on the genetics, but the geneticist thinks it is fine, I would usually side with the expert, once I have understood the matter myself…).
    Although this is a real black box for so many people, I can assure you that most people take any role they play very seriously and do their job extremely well. The thing I was struck most by when I came to NPG, was how well and fair the entire review process worked. It is an impressive machine. Manuscripts fall through the cracks all the time, but all in all, it works pretty well considering the subjective nature that is inherent to the system!
    Feel free to ask questions because there is no intention to keep this process secret, except for the identities of the reviewers! If you want additional information, or a different perspective from another editor, there is a great forum here in which you can ask questions and get multiple answers from multiple editors.


  6. S. Rivlin Says:

    The issue of membership on EBs goes beyond review of manuscripts for the sake of preserving the peer-review system. For many EB members it is first and formost a feather in their cap. It is, of course, also a feather in the cap of the journal that manages to entice a big wig to become a member of its EB. I know of several scientists who are members of several EBs (some are members of 5-10 EBs) and I doubt they could review manuscripts submitted to these journals on a regular basis; they would have no time left for anything else. I myself, am a member of one journal’s EB, but have never received even one manuscript from this journal to review. On the other hand, I regularly review manuscripts for approximately 10 journals at an average rate of one manuscript per week over the past 15 years. I spent anywhere from 5 to 10 hours per manuscript.


  7. bsci Says:

    I’m not sure you were responding to my earlier comment, but I think we agree here. You described how the editorial processes works with a good editor. If problems arise in the process such as those that DrugMonkey implies might happen, it’s more an issue of there being a bad editor and less an issue of the existence of an editorial board.


  8. Peripheral to this discussion of the review process, EBs also play a huge role in launching new journals. When invited to be a EB member, one wants to (or should) feel pride in the new enterprise and submit their best recent work to help raise the journal’s stature.


  9. DrugMonkey Says:

    1. Authors always give a list of those investigators who they would rather not review their work. These lists are always honored.
    Always? Hmm. This seems like a very bad way to do things. Or perhaps a very revealing confession with respect to your journal.
    What I’d like to see is a statement that the lists will be taken into consideration, not that they will be “honored”. Why? Well, it strikes me that it would be pretty damn easy for a PI to supply a list of reviewers to ban that would almost guarantee a super friendly review. no?
    I can think of many relatively “pure” debates in science where the two sides would likely go after each other hammer and tongs on reviews. With legitimate points, not just biased sniping. Sure, the editor should recognize the antagonism but if we rule out these reviews, doesn’t this mean that overall the reviewing will be of lesser quality?


  10. Noah Gray Says:

    Once again from the DrugMonkey, a bit too much conspiracy theory going on for my tastes. There is no reason to ask an author for a list of those people s/he feels cannot be impartial regarding the work if it will not be honored, so the “telling” piece of information that I have revealed escapes me. It is my understanding that all journals allow authors to make such requests. As an author, I would most certainly be pissed if I found out that particular journals did not take that seriously. In the over 600 papers that I have read since coming to the job, I have come across an exclusion list longer than 5 names only 2-3 times. If I can’t find good, tough, fair reviewers for a particular manuscript outside of the 5 people the authors suggest, then I shouldn’t be in this line of work.
    Removing bias, even potential bias, from the review process is part of my job. Thus, I employ the authors to help me eliminate negative bias (and positive bias as well, by examining their acknowledgments section). Does bias still creep in. Yes. Can it typically be neutralized by having more than one reviewer on a paper? Almost always. Are poor decisions still made? Certainly, but that’s why we have an appeals process…


  11. DrugMonkey Says:

    As an author, I would most certainly be pissed if I found out that particular journals did not take that seriously.
    I am not saying journals should ignore these lists. Not at all. I am saying that this should be a mechanism solely to draw the editor’s attention to possible conflicts. Where I draw the line is issuing a rock-solid guarantee that the authors’ wishes will be 100% honored.
    If I can’t find good, tough, fair reviewers for a particular manuscript outside of the 5 people the authors suggest, then I shouldn’t be in this line of work.
    My views are possibly somewhat limited by being close to a number of subfields in which the best possible groups to review any number of papers compromises a list of about 5 or so. I suspect even in larger fields my points have merit though.
    I am not suggesting you can’t drum up some perfectly good reviews outside the short list. But if you want to tap the people going hammer and tongs with up to date stuff in a given area, the list of top experts gets really short in a hurry.
    There is a related consideration which is that using the request for blocking certain obvious reviewers may increase the chance of a softballer reviewer getting selected even if there is no legitimate beef with someone on the ban list.
    And as I said, sometimes it is the case that two labs are bashing back and forth in a good way, and it might be valuable to get the viewpoint of the “countering” lab, even if you have to discount for competitive reasons.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: