Put up or shut up time, all ye OpenSciencePostPubReview Waccaloons!
October 24, 2013
PubMed Commons has finally incorporated a comment feature.
NCBI has released a pilot version of a new service in PubMed that allows researchers to post comments on individual PubMed abstracts. Called PubMed Commons, this service is an initiative of the NIH leadership in response to repeated requests by the scientific community for such a forum to be part of PubMed. We hope that PubMed Commons will leverage the social power of the internet to encourage constructive criticism and high quality discussions of scientific issues that will both enhance understanding and provide new avenues of collaboration within the community.
This is described as being in beta test version and for now is only open to authors of articles already listed in PubMed, so far as I can tell.
Perhaps not as Open as some would wish but it is a pretty good start.
I cannot WAIT to see how this shakes out.
The Open-Everything, RetractionWatch, ReplicationEleventy, PeerReviewFailz, etc acolytes of various strains would have us believe that this is the way to save all of science.
This step of PubMed brings the online commenting to the best place, i.e., where everyone searches out the papers, instead of the commercially beneficial place. It will link, I presume, the commentary to the openly-available PMC version once the 12 month embargo elapses for each paper. All in all, a good place for this to occur.
I will be eager to see if there is any adoption of commenting, to see the type of comments that are offered and to assess whether certain kinds of papers get more commentary than do others. All and all this is going to be a neat little experiment for the conduct-of-science geeks to observe.
I recommend you sign up as soon as possible. I’m sure the devout and TrueBelievers would beg you to make a comment on a paper yourself so, sure, go and comment on some paper.
You can search out commented papers with this string, apparently.
has_user_comments[sb]
In case you are interested in seeing what sorts of comments are being made.
DrugMonkey 
October 24, 2013 at 8:04 am
PLoS One tried this, and the comments there are dead. It would be more interesting if anonymous comments were allowed. But that would invite idiocy.
eLife journal (http://elife.elifesciences.org) is a more interesting experiment. It publishes the reviews and editor letters along with the paper.
LikeLike
October 24, 2013 at 9:15 am
Why on earth would this experiment work when every other version of it so far has failed?
It might work if they allow anonymous, moderated comments. But even then, does anyone really have time to post online comments when we all know how busy scientists are?
Oh, wait a minute…
LikeLike
October 24, 2013 at 9:17 am
I commented on this over at RW. Posters there getting a boner about it, but apprently not realizing that post-publication commenting is non-existent at journals that have tried it. Ghost-town. Allowing anonymous comments would be a disaster.
@TOD: I have always liked what EMBOJ are doing with publishing the peer review files. I wasn’t aware another journal (eLife) was doing it.
LikeLike
October 24, 2013 at 10:28 am
@Dave, part of the problem is most of the journal-specific commenting systems strictly forbid anything that might come across as “OMG they faked teh dataz!” Taking a look at PubPeer et al., that’s what >90% of the comments appear to be about, so it’s hardly surprising that the journal comment spaces are dead when they exclude >90% of their potential traffic. Thankfully PMCommons has taken more of an anything goes approach, so it will be interesting to see how this shakes out.
One thing I’m not keen on, is having to log in just to view the comments. No problem with logging in to comment, but just to read the fuckers? C’mon! PubMed used to be one of the few sites that didn’t require any kind of login to get most of the functionality, so I’m wondering if PMCommons is the first of many services to be rolled out which require login, with the eventual goal being for anyone who uses PubMed to be logged in (and therefore tracked).
The reasonable person in me wants to believe that 10 years from now, a statement like “how can you cite this paper in your grant because according to our records you haven’t even read the abstract” would be impossible. Given the events of this summer involving NSA, such a future would not surprise me any more.
/tin foil hat
LikeLike
October 24, 2013 at 11:09 am
I admire the idea but I am not really optimistic about it.
If this is going to work out, those comments would have to be serious ones. Writing serious comments on serious scientific papers takes a correspondingly serious amount of time – and all the incentives are stacked towards spending that time somewhere else (writing your own papers and grants).
Add to that the fact that a lot of papers will simply not get read by many people despite how good they might be simply because of where they are published.
My bet is on it ending up like the comments sections on PLoS ONE.
LikeLike
October 24, 2013 at 11:34 am
I’m not all that familiar with PubMed (I’m in a slightly different field), but in almost all of the papers on the “has_user_comments” list, I can’t actually find the comments. Are the comments not publicly viewable, or is it just my brain / user interface snafu?
LikeLike
October 24, 2013 at 11:38 am
@LM:
Apparently they are closed to users only for right now:
from: http://www.ncbi.nlm.nih.gov/pubmedcommons/faq/#seecomments
“Why can’t I see any comments in PubMed?
Only PubMed Commons participants can see comments during the closed pilot phase.
Participants need to be signed into their My NCBI account to see and post comments. For information about My NCBI, click here or on the “Sign in to My NCBI” link at PubMed at the top of the page on the right.”
LikeLike
October 24, 2013 at 11:54 am
Comments on journal websites were always doomed because it’s not a central site…you have to be in one place to comment on a PloS paper and another to comment on a Nature paper or whatever. It’s just too much hassle. Centralizing is the best approach. Having to log in, however….is not.
Still, I think that many people will like the opportunity to influence how other people see a given publication, and this is the bully pulpit. I think it will produce some interesting exchanges.
LikeLike
October 24, 2013 at 12:56 pm
DJMH: “Having to log in, however….is not. ”
Yeah, I have a feeling the NCBI is going to limit access for a bit to see how it goes with “qualified commentators”. I’ve done a few myself so far. I see it as a nice place where I can keep “article summaries” of useful papers conveniently online more than much else- some of which are critical; some of which are “thank you so much for writing this, now I don’t have to, oh and do you think such&such is a plausible mechanism of tumor infiltration?”.
LikeLike
October 24, 2013 at 1:53 pm
Interesting. I hadn’t noticed y0u have to be logged in just to see the comments. That seems to miss the point. I mean, isn’t part of the dealio here to warn off the relatively naive user? To adjust the expectations based on the simple reading of the abstract?
Hmmm. I hope they open that up to all users of the PubMed records.
LikeLike
October 24, 2013 at 2:40 pm
DM:
Hmmm. I hope they open that up to all users of the PubMed records.
Are you sure you want that?
Have you ever seen what kind of comments on scientific articles are published on Uncommon Descent and other places of the sort?
If anyone can comment, the place will end up looking like the comment sections of the popular websites – every articles about evolution, vaccines, GMOs, etc. will be completely overrun by comments from various sorts of crazies.
I wouldn’t let anyone who doesn’t have at least a few entries in PubMed to his own name comment.
Everyone should be able to read comments though, requiring you to be logged in is stupid. Even more so given how the system likes to log you off automatically every few hours despite you checking the “Remember me” box every time you log in
LikeLike
October 24, 2013 at 5:43 pm
I agree with all comments from above and being a faculty of F1000 I can also predict this variation:
” I really enjoy this outstanding manuscript by one of the leaders in the field. This is how science should be done….. Oh yeah he/ she is on my study section.
Signed sycophantus maximus ps pls remember this when you read my grant”
LikeLike
October 25, 2013 at 5:37 am
“Writing serious comments on serious scientific papers takes a correspondingly serious amount of time – and all the incentives are stacked towards spending that time somewhere else (writing your own papers and grants).”
“I really enjoy this outstanding manuscript by one of the leaders in the field. This is how science should be done….. Oh yeah he/ she is on my study section.”
These are exactly the reasons why this experiment in the social media-ization of science will fail like all the other ones.
We already have serious comments. They are called “review articles.”
Look. Science is complex. You can’t make a valuable contribution by shooting your mouth off without thinking. You have to think, and think hard. You have to painstakingly put together evidence (from the literature and/or your own experiments) to come up with an idea and support it. This is inherently a slow process. And when you do it, you need to get credit for it – that’s just how the system works. So unless you can use your online comment activity to support your bids to get grants, promotion and tenure (which is something that will never happen), you aren’t going to put enough effort into it to make it worthwhile for you or anyone else.
LikeLike
October 25, 2013 at 6:08 am
Very interesting–almost everyone predicts either a comment ghost town, or a horror show composed of flat earthers and trolls. Self-fulfilling prophecies?
LikeLike
October 25, 2013 at 6:32 am
@Grumble, Re: people simply not having enough time to deal with this.
If I read a paper and it’s a pile of horse shit, and is by my competitor who is fighting for money at the same study section as me, darn right I will trash that sucker. If there’s anything “weird” about the data (as for the majority of cases listed at PubPeer), darn right I will trash that sucker. If that person’s next grant gets dinged because they had to retract/correct some stuff, that increases my chances of getting funded and is most certainly worth the relatively minor investment of my time. Not to mention the satisfaction of removing a leech from the funding pool.
Does not putting up with crap science in my field make me a bad person?
LikeLike
October 25, 2013 at 6:40 am
This will fail.
We already comment on papers all the time. Everyone knows what the crap is and who generates it in their subfield of interest. Everyone discusses the important stuff.
I personally see no reason for there to be some kind of public online record of these discussions, and apparently neither does anyone else.
LikeLike
October 25, 2013 at 7:43 am
Clearly will fail. There is no incentive to comment and many disincentives. Simple economics.
Posting peer reviews and editorial comments online post-publication could be very useful, however, so its good to see the Embo and elife models taking off.
LikeLike
October 25, 2013 at 9:39 am
@Ola – Of course intolerance of bullshit does not make you a bad scientist – who said it does?
But your comment exactly proves the points against comments-for-papers. The reasons you give for trashing a crap paper sound to me like vindictiveness plus eeking out some professional advantage. This has little to do with the reasoned discourse that truly moves science forward.
On top of that, let’s consider two scenarios, non-anonymous and anonymous comments. If comments are non-anonymous, are you really going to go on record to trash your peer’s work? Just think about the implications. And if they are anonymous, is anyone reading them going to take them as anything more than… someone being vindictive and trying to eek out a professional (or some other) advantage? Would such comments help YOU to evaluate a paper you encounter on PubMed?
LikeLike
October 25, 2013 at 10:00 am
If some anonymous person says “hey, the Western in Figure 5a is exactly the same as the one in Bong et al 2005 Fig 2c” then yeah that can help you to evaluate. it doesn’t matter one whit that the person was ‘vindictive’ because you can evaluate that claim for yourself.
Same deal if someone says “yeah but we published almost the same thing five years ago in Journal Obscura-ology”.
the only anonymous comments that require discounting are the ones that offer some opinion or criticism that is unverifiable. and sure, in those cases you discount accordingly. (although I’m not sure some real-namer opinion is any more valuable unless they are claiming some data that they could then send you or otherwise publish)
LikeLike
October 25, 2013 at 10:45 am
I’ll operate under the assumption, for now, that it’s not that common (0.01% of papers? Less?) for a reader to pick up obvious evidence of fraud.
As for opinions and criticisms: in science these are both a dime a dozen. They are far more valuable than when they are well thought through and well-supported. That is why you are unlikely to encounter anything of value in the comments section.
LikeLike
October 26, 2013 at 7:20 am
I can think of at least one incentive to comment. You can refer to your own papers in your comment. If you already published something on the topic or if in the future you build on the topic you can go back and say how it influenced you.
Let’s say that in the future comments are visible to all when viewing an abstract. Even a short comment like “this paper showed this, which led us to do this cool thing, see here”. Could help raise the profile of your papers especially if you’re commenting on popular papers.
It’s reasonable that that will increase readership of your papers and ultimately citations.
LikeLike
October 27, 2013 at 9:35 pm
The naysayers can sneer all they want at “waccaloons”, but having Pubmed Commons is far better than not having it. In just a short while the commentary at Faculty of 1000 has already become a valuable resource. This is that writ large. No, it’s not perfect, it’s no silver bullet, but cripes, I don’t understand the grumbling and the impulse to tear it down that is infesting these comments. Don’t like it? Try to improve it or get out of the way.
LikeLike
October 28, 2013 at 3:16 pm
This is that writ large.
No, it is not. The Faculty of 1000 recommendations are all subject to editorial review before publication.
LikeLike
October 30, 2013 at 12:41 pm
I hope it can be a forum for open dialog. I can ask questions like, “We tried to reproduce this but only measured 20 instead of 200 as in your paper. Can you give more details on how to perform step 3?” Having questions in the open will give more incentive for authors to help others and support reproducibility.
LikeLike
January 24, 2014 at 1:18 pm
Hi there to all, the contents existing at ths weeb page are genuinely aweome for people knowledge,
well, keep up the nice work fellows.
LikeLike