Reviewing academic papers: Observation
April 11, 2012
When you are reviewing papers for a journal, it is in your best interest to stake out papers most like your own as “acceptable for publication”.
If it is a higher IF than you usually reach, you should argue for a manuscript that is somewhat below that journal’s standard.
If it is a journal in which you have published, it is in your interest to crap on any manuscript that is lesser than your typical offerings.
DrugMonkey 
April 11, 2012 at 7:27 am
So root for the underdog to make it to the big leagues and shit on the competition at your level?
LikeLike
April 11, 2012 at 7:32 am
Exactly.
LikeLike
April 11, 2012 at 7:46 am
I like it.
LikeLike
April 11, 2012 at 7:48 am
So much for unbiased review, eh? I am all in favor of supporting the work of my colleagues within my own field, regardless of the journal title. However, I would be as equally critical of their work as I would anyone else. The point is to prevent un-reproducible pieces of shit from coming out in print.
LikeLike
April 11, 2012 at 7:49 am
I’m also sure there are a series of principled strategies for getting more tapioca at lunch time in a nut house. The signal to noise ratio in peer review is near zero.
LikeLike
April 11, 2012 at 7:49 am
Worst. Reviewer. Ever.
LikeLike
April 11, 2012 at 9:31 am
May I assume that you are not referring to
Noah?
LikeLike
April 11, 2012 at 9:53 am
I disagree with Noah in that there can be much worse reviewers, but this is pretty bad. In what reality should one’s standard be that the worst article for a specific journal should be one’s own?
The substantive comments on research quality should be identical regardless of journal. An article with avoidably bad methodology or interpretation shouldn’t be published anywhere.
The only place reviews should differ based on journal are on the importance and strength of findings. (i.e. a weak effect where the authors acknowledge not-easily-avoided methodological weakness is ok for some lower tier journals).
For reviewing borderline manuscripts, my own approach is to highlight the manuscript importance and acknowledged weaknesses and let the editor decide whether or not those are appropriate for the specific journal.
LikeLike
April 11, 2012 at 3:56 pm
Is this why there seems to be a “club” of people who are invited to review for the higher-tier journals – consisting mainly of people who themselves publish in those journals?
Whatever the cause, the result is an incestuous clusterfuck. And that clusterfuck could be the reason why, by some estimates, 94% of “high-impact” studies are irreproducible.
LikeLike
April 11, 2012 at 4:09 pm
Sorry, 89%, not 94%.
Of course, if the reviewers of articles submitted to high tier journals have a vested interest in letting only the best science be published there (because that’s where they themselves publish), then, theoretically, this should result in better science in higher-tier journals.
What really might happening, though, is that reviewers confuse “exciting” with “better”, both in their reviews and in their own submissions. So, they get their irreproducible crap (and that of others) into Nature by compiling (and enthusiastically reviewing) nice just-so stories, not by rigorously examining the data for flaws and excluding questionable results.
LikeLike
April 12, 2012 at 12:28 am
Grumble,
I agree
LikeLike
April 12, 2012 at 2:53 am
Of course there is a club Grumble and of course one of the things they do is enforce a sort of technical and lab-size threshold which limits the competition. They *also* drive the ship on IF and Glamour as (circularly) defining the “best” science. This has downstream effects on who gets the grant money, the creation of mega-labs, the exploitation of postdocs under sweatshop conditions, etc.
LikeLike
April 12, 2012 at 5:28 am
…And maybe it also has effects on the quality of the science. If the goal at the outset isn’t to figure out how a biological process works, but rather to construct a shiny, exciting and “complete” story for other members of the club to jerk off to, then finding out how the biological process works will take a back seat. If that leads to 89% of Nature and Science papers being flat out wrong, there is a BIG problem with the social structure of science. Even if that number is inflated by two or three fold, it’s still a huge problem.
LikeLike
April 12, 2012 at 5:56 am
If the goal at the outset isn’t to figure out how a biological process works, but rather to construct a shiny, exciting and “complete” story for other members of the club to jerk off to,
This is indeed the problem. The focus is on the “get” rather than meaningful scientific advance. Things being likely to be true are a mere inconvenient hurdle in the process.
Ask yourself how often someone is lauded for “getting (a) Nature paper(s)” over the specific content of those papers….for values greater than zero we’re doing it wrong.
LikeLike
April 12, 2012 at 7:28 am
How science is and how science ought to be are two different things.
My guess is that DM’s Worst.Reviewer.Ever is far more prevalent than anon #1’s “I would be as equally critical of their work as I would anyone else.”
LikeLike
April 17, 2012 at 12:16 pm
Great game theory! Thank you =)
LikeLike