Reviewing academic papers: Observation

April 11, 2012

When you are reviewing papers for a journal, it is in your best interest to stake out papers most like your own as “acceptable for publication”.

If it is a higher IF than you usually reach, you should argue for a manuscript that is somewhat below that journal’s standard.

If it is a journal in which you have published, it is in your interest to crap on any manuscript that is lesser than your typical offerings.

Advertisements

No Responses Yet to “Reviewing academic papers: Observation”


  1. So root for the underdog to make it to the big leagues and shit on the competition at your level?

    Like

  2. DrugMonkey Says:

    Exactly.

    Like

  3. anon Says:

    So much for unbiased review, eh? I am all in favor of supporting the work of my colleagues within my own field, regardless of the journal title. However, I would be as equally critical of their work as I would anyone else. The point is to prevent un-reproducible pieces of shit from coming out in print.

    Like

  4. miko Says:

    I’m also sure there are a series of principled strategies for getting more tapioca at lunch time in a nut house. The signal to noise ratio in peer review is near zero.

    Like

  5. Noah Gray Says:

    Worst. Reviewer. Ever.

    Like

  6. drugmonkey Says:

    May I assume that you are not referring to

    If it is a journal in which you have published, it is in your interest to crap on any manuscript that is lesser than your typical offerings.

    Noah?

    Like

  7. bsci Says:

    I disagree with Noah in that there can be much worse reviewers, but this is pretty bad. In what reality should one’s standard be that the worst article for a specific journal should be one’s own?

    The substantive comments on research quality should be identical regardless of journal. An article with avoidably bad methodology or interpretation shouldn’t be published anywhere.
    The only place reviews should differ based on journal are on the importance and strength of findings. (i.e. a weak effect where the authors acknowledge not-easily-avoided methodological weakness is ok for some lower tier journals).

    For reviewing borderline manuscripts, my own approach is to highlight the manuscript importance and acknowledged weaknesses and let the editor decide whether or not those are appropriate for the specific journal.

    Like

  8. Grumble Says:

    Is this why there seems to be a “club” of people who are invited to review for the higher-tier journals – consisting mainly of people who themselves publish in those journals?

    Whatever the cause, the result is an incestuous clusterfuck. And that clusterfuck could be the reason why, by some estimates, 94% of “high-impact” studies are irreproducible.

    Like

  9. Grumble Says:

    Sorry, 89%, not 94%.

    Of course, if the reviewers of articles submitted to high tier journals have a vested interest in letting only the best science be published there (because that’s where they themselves publish), then, theoretically, this should result in better science in higher-tier journals.

    What really might happening, though, is that reviewers confuse “exciting” with “better”, both in their reviews and in their own submissions. So, they get their irreproducible crap (and that of others) into Nature by compiling (and enthusiastically reviewing) nice just-so stories, not by rigorously examining the data for flaws and excluding questionable results.

    Like

  10. anon Says:

    Grumble,

    I agree

    Like

  11. Drugmonkey Says:

    Of course there is a club Grumble and of course one of the things they do is enforce a sort of technical and lab-size threshold which limits the competition. They *also* drive the ship on IF and Glamour as (circularly) defining the “best” science. This has downstream effects on who gets the grant money, the creation of mega-labs, the exploitation of postdocs under sweatshop conditions, etc.

    Like

  12. Grumble Says:

    …And maybe it also has effects on the quality of the science. If the goal at the outset isn’t to figure out how a biological process works, but rather to construct a shiny, exciting and “complete” story for other members of the club to jerk off to, then finding out how the biological process works will take a back seat. If that leads to 89% of Nature and Science papers being flat out wrong, there is a BIG problem with the social structure of science. Even if that number is inflated by two or three fold, it’s still a huge problem.

    Like

  13. Drugmonkey Says:

    If the goal at the outset isn’t to figure out how a biological process works, but rather to construct a shiny, exciting and “complete” story for other members of the club to jerk off to,

    This is indeed the problem. The focus is on the “get” rather than meaningful scientific advance. Things being likely to be true are a mere inconvenient hurdle in the process.

    Ask yourself how often someone is lauded for “getting (a) Nature paper(s)” over the specific content of those papers….for values greater than zero we’re doing it wrong.

    Like

  14. neuromusic Says:

    How science is and how science ought to be are two different things.

    My guess is that DM’s Worst.Reviewer.Ever is far more prevalent than anon #1’s “I would be as equally critical of their work as I would anyone else.”

    Like

  15. Shlogbaum Says:

    Great game theory! Thank you =)

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: