Post-publication peer review and preprint fans
June 6, 2013
Anyone who thinks this is a good idea for the biomedical sciences has to have served as an Associate Editor for at least 50 submitted manuscripts or there is no reason to listen to their opinion.
DrugMonkey 
June 6, 2013 at 6:24 am
Not disputing you, because I don’t have enough information to, but care to elucidate? It’s often foolish to dismiss people’s opinions simply because they are not a member of the “correct” class. I am not sure this is one of those cases.
LikeLike
June 6, 2013 at 6:36 am
why?
LikeLike
June 6, 2013 at 6:57 am
Why does it work in physics (when it wouldn’t in whatever we do)? Or does it not work in physics?
LikeLike
June 6, 2013 at 7:04 am
troll.
LikeLike
June 6, 2013 at 8:21 am
why?
I think the biggest fans of this nonsense are delusional about the quality of work people are willing to offer up as their initial presentation. Also the degree to which they will improve their initial drafts in the absence of a strong inducement to do so.
LikeLike
June 6, 2013 at 8:42 am
An interesting article talking about the interaction between high profile journal publishing & arcXiv in physics: http://www.aps.org/publications/apsnews/201211/preprint.cfm
It does show that the physicists worry about embargos/undermining their publication in Nature/Science, but that a prime incentive to pubish preprints in arcXiv is that priority is established through arcXiv publications.
I think the assumption that preprint publications would stay the same if they were published in a generally available site v being sent confidentially to an editor is wrong. Presumably the articles would have to be better if everyone could read them, and the incentive of having people laugh at you and publicly pick holes in your work would play some role in the quality provided. I agree that there needs to be an incentive — it appears that in physics, the incentive is provided by the credit for priority. Not sure if that can work in fields that are more wishy washy, but it’s not an idea to dismiss out of hand.
I am starting to develop a sneaky suspicion, worrisome, really, that there might be too much biomed science, though, and that’s part of the problem. Potentially pre-print publication might not work in bio for the same reason that people struggle so mightily to get into glamor journals as a mark of the quality of the work: the work may have become so siloed that only a few people have the ability to judge work on its merits and most are relying on secondary judgment.
LikeLike
June 6, 2013 at 8:51 am
Well, there’s this:
http://www.nature.com/nature/journal/v497/n7450/full/497433a.html
About 75-90% irreproducibility in what’s alleged to be the best of the best. I think that would be a good reason to check what’s been published.
LikeLike
June 6, 2013 at 12:48 pm
I post on arXiv to establish priority. I submit to journals to get the “seal of approval” (however much you think that’s worth).
Also, the infamous “in preparation” or “under review” listings on the CV have a bit more weight when there’s an arXiv link to back it up. The seal of approval still matters, but at least you know the “under review” listing isn’t just a bluff.
LikeLike
June 6, 2013 at 12:48 pm
BTW, once it’s on arXiv an unethical reviewer can’t beat my claim of priority.
LikeLike
June 6, 2013 at 3:36 pm
You’re a fucken crybaby. Sacke the fucken fucke uppe and edit that shitte and quit fucken whining.
LikeLike
June 6, 2013 at 8:06 pm
Are you talking to me or DM?
LikeLike
June 7, 2013 at 9:24 am
@zb
Peer review is more important in experimental sciences because we can’t really rely on mathematical proof as things like physics and math do — it’s no coincidence that the fields that are into preprints are things like physics, math, and computer science. It is a lot easier to fudge (either by intention or incompetence) an experiment than a proof.
LikeLike
June 7, 2013 at 9:54 am
“Peer review is more important in experimental sciences because we can’t really rely on mathematical proof as things like physics and math do”
I think there is a possibility of a difference in content/method making arcXiv not a suitable model for biology, but I’m not convinced that the math explains the difference. First, though the math-based sciences might be amenable to having a paper more completely represent the idea (i.e. you show the math proof, and it’s all there, and nothing depends on whether a grad student accidentally dropped an extra ml of solution into something), math still needs to be reviewed by peers, since not everyone will be able to tell that the math is right. Second, if math was the answer, all theory based fields should be moving to arcXiv model, and I don’t think they are (but maybe I’m wrong?).
I still think the difference is the size of the peer group that thinks it can fairly assess content. So, though the math might not be understandable to everyone in a theoretical physics field, there are enough in the group who feel confident assessing a paper for themselves. In bio, that group may be too small a subgroup of the people who are interested in the paper. Otherwise, how could it be wrong that more reviewers would be a good thing, rather than a bad one?
LikeLike
June 7, 2013 at 10:59 am
@zb
Oh, I agree that the size of the peer group matters, but I’m not sure that’s not just saying the same thing in a different way — the thing about math is that the application field often has little to do with the math itself — things like game theory can apply in fields as different as economics and evolutionary theory and in principle, an expert in one field could be a useful reviewer for a manuscript from another. Whereas in experimental biology, even people in related subfields may use completely different techniques and be unable to truly judge a paper from outside their subfield. And it does seem that most theory-based fields are moving to arXiv or their own systems of preprint distribution. Preprint enthusiasts like to point to an editorial by Krugman where he mentions that economics pretty much abandoned journals for preprints in the 1980s — so even before physics.
But as for “how could it be wrong that more reviewers would be a good thing, rather than a bad one” in regard to open peer review of preprints, I don’t think anyone really thinks it would be a bad thing– just that they doubt that anyone would bother to spend their time to comment, or that authors would really change their papers based on a comment alone. There’s a comment feature at PLOS ONE that is intended for post-publication peer review, but hardly any papers get comments.
LikeLike
June 7, 2013 at 4:44 pm
One issue I have with the pre-print concept is that it doesn’t generally focus on the flaws. The general idea is that it’s harmless for people to put manuscripts out. They’ll either be good and get read or be bad and get ignored. There is a rather infamous preprint that doesn’t usually get labelled as such. Economics is also a preprint focused field. This is partially because their peer review process has gotten so dysfunctional that it can take years from first submission to publication. The field has prestigious preprint sites, like NBER, where already famous economists can put their unreviewed articles (Let’s skip the separate issue of a prestigious site that you can only use if you’re already in the in crowd). One such article was about the relationship between debt and GDP by Reinhart and Rogoff. Despite never getting peer reviewed and receiving a lot of negative post-publication peer review, it became accepted as accurate amount many policy circles and possibly influenced government actions. It also turned out that they made some serious mistakes, some of which could probably have been caught in a basic peer preview process.
I wonder if some of the unstated reasons preprints have worked in math & physics is that mistakes just aren’t that dangerous. Preprint/post peer review of studies with direct clinical relevance could be dangerous.
LikeLike
June 9, 2013 at 4:37 am
Tend to agree that biomed isn’t ready for preprint servers because of the reasons pointed out by drugmonkey above. You should just take a look at some of the stuff pointed out on PubPeer for data that was reviewed and published in a “journal”.
However, I think we can get there one day. One way to accelerate that is by commenting on already published material. It seems logical that this will eventually become more and more adopted within the community and once it’s the norm then it doesn’t matter where an article is published. It will only matter what the field as a hive community says about it.
Attending journal clubs at 3 major research universities on two continents I have never once left a journal club in which it was concluded that everything was fine with the paper. There are comments on every paper, some major and some minor. These comments should appear for others to see and to give a real “impact factor” of the findings. It’s easy to write a journal club recap on PubPeer.
If it’s not obvious, we created PubPeer.com for exactly this reason.
LikeLike
June 9, 2013 at 10:23 am
One way to accelerate that is by commenting on already published material. It seems logical that this will eventually become more and more adopted within the community and once it’s the norm then it doesn’t matter where an article is published.
Why does it “seem logical” that–despite the fact that post-publication article commenting has been available for years and has never caught on–commenting on articles will eventually become “the norm”? What seems logical to me is that scientists will continue to not comment on published articles on the Web, just as they have been doing so far.
LikeLike
June 9, 2013 at 10:32 pm
What I mean is that their is either a lack of things to say about publications or some sort of energy barrier keeping us from writing these things down for others see and use. In my experience it’s very clear that it’s not the former. Somebody will eventually figure out a solution to the latter and commenting will eventually be the norm.
You seem to have no problem spending some time on a Sunday writing down public comments about whether or not people will eventually come around to writing down public comments on articles. Why do you spend time on this? Do you also spend time writing a few sentences about an article you read recently? I”m just curious what the energy barriers are that are keeping people from writing post-publication comments.
LikeLike
June 10, 2013 at 2:00 am
The “energy barriers” are multiple. (1) There are already multiple venues in which one can discuss the literature with one’s colleagues: journal clubs, lab meetings, conferences and symposia, lunch, etc. (2) There is a strong incentive for these discussions to remain ephemeral, and not to be rendered permanent, searchable, and attributable. Are you gonna leave a comment on Dr. Professor Nobel Prize Winner’s latest Science paper explaining how boring, stupid, and poorly controlled the experiments are? (3) In the cases where such discussions are not ephemeral–namely, formal published review articles–scientists get scientific and academic credit for those reviews, since they are real publications that go on the CV.
You are delusional if you think that these obstacles are technical ones that can be overcome by some kind of “ghee whiz” improvement to commenting functionality on journal Web pages or by the creation of third-party Web pages for post-publication commenting. These obstacles reflect powerful structural features of the scientific enterprise which make it an all-down-side no-upside proposition for scientists to post written comments to published scientific articles on the Web.
LikeLike
June 10, 2013 at 6:33 am
@CPP, the major obstacle for PLos One and commenting systems on other journal websites is they require a real identity. Big surprise, when critiquing others’ work some people don’t like using their real IDs. Also read their commenting terms/conditions very carefully — you can’t make comments that would be deemed allegations of misconduct. Going onto PLoS One and saying “that there blot looks like it’s a fukken copy of that there other blot” is just not possible.
In contrast, PubPeer enables peers to comment anonymously, but you have to register using a real ID which is linked to the fact you have authored at least one paper. Of course, there’s always the security risk that the owners of the site could reveal your ID if they were subpoenaed to do so. My pet theory is that PubPeer is actually a phishing site set up by the NSA and the ORI to catch whistle-blowers.
/tin foil hat.
LikeLike
June 10, 2013 at 6:50 am
Dunno about other areas of the natural sciences, but pubpeer has zero chance of making any sort of impact on the biological and biomedical sciences. While goofy science enthusiasts may get off on masturbating over papers there, actual scientists who are engaged in contributing to bioscientific knowledge are never ever in a million years going to spend time and effort on anonymous internet commenting on published papers.
LikeLike
June 10, 2013 at 9:18 am
actual scientists who are engaged in contributing to bioscientific knowledge are never ever in a million years going to spend time and effort on anonymous internet commenting
So what are you contributing to? 🙂
LikeLike
June 10, 2013 at 9:23 am
I am contributing to pointing out that you are a disingenuous misquoting asshole who left out the key phrase “on published papers”.
LikeLike
June 10, 2013 at 11:42 am
In contrast, PubPeer enables peers to comment anonymously, but you have to register using a real ID which is linked to the fact you have authored at least one paper.
No need to create an account on PubPeer. You can submit a comment without signing in.
(1) There are already multiple venues in which one can discuss the literature with one’s colleagues: journal clubs, lab meetings, conferences and symposia, lunch, etc.
Search committees, tenure committees, NIH study sections, etc. don’t necessarily hear your journal club and lab meeting conversations. You tear apart a paper in journal club and the first author still gets a job because you never influenced anyone other than your journal club members with your comments.
(2) There is a strong incentive for these discussions to remain ephemeral, and not to be rendered permanent, searchable, and attributable. Are you gonna leave a comment on Dr. Professor Nobel Prize Winner’s latest Science paper explaining how boring, stupid, and poorly controlled the experiments are?
Sure. If it’s anonymous why not? It will do science some good and allow him and other colleagues to respond.
(3) In the cases where such discussions are not ephemeral–namely, formal published review articles–scientists get scientific and academic credit for those reviews, since they are real publications that go on the CV.
Why get rid of these avenues? Although they do take much more time than writing down some journal club comments on PubPeer.
LikeLike
June 10, 2013 at 1:07 pm
Dude, you really don’t get it, do you? Real scientists don’t write on-line comments to published research manuscripts because there is no incentive for them to do so. It is neither fun nor professionally rewarding. And it is never going to be.
I have looked at comments on Pubpeer, and the overwhelmingly vast majority of the ones on biosciences papers involve obsessive deconstruction of western blot lanes, image duplication/manupulation, and other signs of sloppiness/shenanigans. Sorry, but real scientific discussion by working scientists who actually generate the research literature is never going to occur in non-ephemeral, public, anonymous Internet venues.
LikeLike
June 11, 2013 at 10:05 am
” If it’s anonymous why not? ”
1. Because your identity can be guessed, especially if you comment a lot and make a lot of references to other work.
2. Because science doesn’t actually benefit from a horde of internet trolls saying the first stupid thing that comes to mind regarding every interesting paper that comes out.
3. Because science actually does benefit from well thought-out, well-reasoned, and well-written articles that propose or defend a hypothesis, and/or summarize the literature. Writing this sort of thing takes WORK, and lots of it. So much work, in fact, that it leaves no time for writing well thought-out, well-reasoned and well-written internet comments on all the articles you just read.
4. Because, as CPP said, if one gets credit for writing reviews, but not for internet keyboard diarrhea, no one is going to spend the time and energy required to post thoughtful comments.
So grow up, PubPeer. A good app does not always change the world. Especially if it’s not a good app.
LikeLike
June 12, 2013 at 10:50 am
I don’t get the venom against PubPeer. Everyone here is spending time writing down comments about the biomedical science system. Why? Presumably because you enjoy participating in such conversations, and/or (delusionally or otherwise) hope to influence how others think about these topics.
You don’t necessarily assume that you’ve got to spend hours researching each comment you make, you toss your idea, informed by years of operation within this system, into the ring. No real difference with journal stuff–if we take apart a paper in jclub, why shouldn’t those discussions also occur online? Grumble, your point (1) can be solved by not using the same pseud each time you comment. Easy-peasy. Your point (2)…I don’t get it, do you think DM’s blog, or any science blog you frequent, is populated by science trolls? No? then why would another science commenting site?
What PubPeer initially screwed up the most (the registration and authorship requirements), it has dismantled. The other thing it has failed to do is to publicize sufficiently. The third thing that would help it is if there were a couple of pseud bloggers who would sign on to have their paper comments show up at Pubpeer as well, maybe mirrored or something like that, so there was an instant reason to check out pubpeer. Probably some other things could help it too. But as an author, I would simply love to know what people say about my papers, through the shroud of anonymity. Gives authors the chance to respond, too.
LikeLike