Responding to comments on pre-prints
March 2, 2018
almost tenured PI raised some interesting questions in a comment:
So you want me to submit a preprint so I can get comments that I have to spend time responding to? No thanks. I spend enough time responding to the comments of reviewers from real journals. I can’t imagine how much time I’d have to spend responding to comments that are public and immortalized online. No paper is perfect. How many comments does a paper need before it’s acceptable for publication? Where does it end? I do not need more work when it already feels like the bar to publishing a paper keeps rising and rising.
Ok, so first off we need to recognize that the null hypothesis right now has to be that there will not be extensive substantial commentary on the preprints. PubMed Central shut down its commenting scheme for lack of use. Journals have been trying out various systems for years (a decade or more?) without general success. The fear that each posted pre-print will attract a host of journal-club style comments is probably not well supported.
But lets suppose your preprint does get some critical comment*. Are you obliged to respond?
This ties into the uncertainty and disagreement over when to submit the preprint. At what stage of the process do you post it? Looking at the offerings on bioRxiv, I think many folks are very close to my position. Namely, we are waiting to submit a preprint until it is ready to go out for peer review at a journal.
So any comments it gets are being made in parallel with review and I can choose to address or not when I get the original decision back and need to revise the manuscript. Would the comments then somehow contaminate the primary review? Would the reviewers of a revision see the comments on the pre-print and demand you address those as well as the regular peer comments? Would an Editor? For now I seriously doubt this is a problem.
So, while I think there may be many reasons for people not to want to post their manuscripts as pre-prints, I don’t think the fear that this will be an extra dose of Reviewer #3 is well supported.
__
*I had one get a comment and we ended up including something in a revision to address it so, win-win.
DrugMonkey 
March 2, 2018 at 2:51 pm
“Looking at the offerings on bioRxiv, I think many folks are very close to my position. Namely, we are waiting to submit a preprint until it is ready to go out for peer review at a journal.”
Is that right? What happened to submitting “works in progress,” as you argued in your previous post on this topic.
So turns out you agree with my position on preprints. But you didn’t say that in response to my comment. Instead, you were rude to me …. because I used to be in physics?! What a jackass you are!
LikeLike
March 2, 2018 at 4:22 pm
Where did I argue my personal position on preeps was for works in progress?
LikeLike
March 2, 2018 at 4:52 pm
“waiting to submit a preprint until it is ready to go out for peer review at a journal” — I agree that this is the norm, and it’s roughly what I do. I haven’t gotten any significant scientific comments from preprints.
LikeLike
March 3, 2018 at 12:55 pm
I don’t think the problem is that the pre-print comments are extra comments that needs to be addressed to get published.
But I think having a comment critical of your pre-print (thats somewhat valid but takes too much effort to address) means you’d have a published paper (say a couple of months down the line) that is associated with a public critique that may unduly influence how the paper is perceived by some.
LikeLike
March 3, 2018 at 2:17 pm
I think biologists are doing preprints wrong. People talk about preprints on arXiv, but the idea that they would have a comment section like a blog isn’t there. Granted, part of this might be because arXiV started in the 1990s when such things would have been technically difficult, but I wonder if maybe bioRxiv has a bit more to learn from its inspiration.
LikeLike
March 4, 2018 at 9:39 am
Speaking of the arXiv, they did a massive user survey a few years ago and published the results. One question was whether people wanted comments enabled for uploaded manuscripts. The response was overwhelmingly “no”.
LikeLike
March 4, 2018 at 10:13 am
I just don’t understand the function of pre-print servers. There may be some interesting information in a pre-print article (like what you would see on a poster at a conference), but they’re not citable (at least I never would) and also are not guaranteed to get published. Pre-prints basically just seem to be a way to officially put “manuscript in preparation” on your CV, which is an easy way for youn-guns to pad their CVs for jobs.
LikeLike
March 4, 2018 at 10:26 am
Plus, “ownership” of an idea traditionally comes from the investigator who firsts publishes on it (i.e. Watson/Crick/Franklin’s structure of DNA, Dounda’s CRISPR, Mello/Fire’s RNAi), which has really significant effects on job prospects, promotion/tenure, and funding. Well-established Graybeards who don’t care about these things may be able to absorb these effects, but how many early to mid career investigators are going to be willing to put their 1st, 2nd, or even 3rd tier journal level ideas out there before they’re published and run the risk of getting scooped? It’s easy enough to get scooped without posting pre-prints, so why tie one hand behind your back and shoot yourself in the foot?
LikeLike
March 4, 2018 at 12:43 pm
Philapodia, I agree with you. I am also trying to understand why people do it. It looks like posting to pre-print servers makes people feel like they have been productive. It’s not accepted, but it’s out there. Google scholar finds them and sends notices. Clincians, in particular, often don’t have the patience for the review cycle(s). So, for early-career people who expect a long protracted process of review, they might as well put it out there. Imagine, if you think that the review process is broken, as demonstrated by those damn reviewers always giving you bad comments, then posting pre-prints is a way to feel like you have avoided them… a little.
LikeLike
March 4, 2018 at 1:41 pm
A preprint that anyone can download and read is infinity better than stating a manuscript title and saying it is “in preparation”.
LikeLike
March 4, 2018 at 3:29 pm
“A preprint that anyone can download and read is infinity better than stating a manuscript title and saying it is “in preparation”.”
Only if you’re a newly minted post-doc trying to land your first job and you need to prove that you have done something. Other than that, I think it’s a net negative career-wise and seems kind of desperate.
LikeLike
March 4, 2018 at 6:26 pm
It seems to me that using preprint servers to “prove productivity” comes back to the reputation problem and the issue of shortcuts. Deciding whether a preprint is “real” means that I have to read it! There are many situations where I as a reviewer have neither the time to read nor the qualifications to judge the validity of a scientific paper, but am required to judge the importance of said scientific productivity – think study section or promotion-and-tenure committees. Being able to say “first author on X papers in respected journals” is shorthand for the impossible “I read all your papers in depth and think you are doing great work”. In my view, a preprint is equivalent to an abstract at SFN – infinitely better than nothing – but also infinitely worse than having a published paper. A few comments on this
1. As a member of a study section looking at young grants (F31, F32, K99), I often saw that an SFN abstract could help show productivity is in process. I can see how a preprint could help here, but really only in a desperation move to give evidence as better than “in preparation”. I never saw this help in an R01 study section situation.
1a. What I am currently seeing in study section by the way is that people are listing everything and anything in the new biosketch format, from abstracts to preprints to in preparations. It has become very difficult to actually identify productivity at this point. (I had to go to pubmed to determine productivity for all my reviews this cycle – which can be difficult if the investigators’ name is Smith or Kim.)
1b. One thing that I’ve seen as a PI is that there are students who are able to get the abstract out, but cannot do the work to carry the paper through to publication. These students run into a buzz saw when they get judged on productivity. I wonder where preprints will fall for these students.
2. Proof of productivity is not how I see preprints are being used. Instead, what I consistently see in both ArXiv and BioArXiv is that they are being used to allow communication while fighting with journals over publication. Given the current process (in which a recently published paper at a GlamourMag had a submission date in 2016 and an acceptance date in 2018, or in which we all know papers that have spent six months at one journal to six months at another to…), it is being used to communicate work for discussion. IMHO this is a good idea, but I’m concerned about cleaning up the trail as the results change due to reviewer input.
LikeLike
March 4, 2018 at 6:31 pm
I agree with DM on this.
“In preparation” can be anything from ‘I just started the experiments’ to almost ready to submit.
A preprint clearly shows that you have something finished enough that you think is ok to be put out there and get judged by others in your field.
LikeLike
March 4, 2018 at 7:09 pm
@Qaz
“Instead, what I consistently see in both ArXiv and BioArXiv is that they are being used to allow communication while fighting with journals over publication. Given the current process (in which a recently published paper at a GlamourMag had a submission date in 2016 and an acceptance date in 2018, or in which we all know papers that have spent six months at one journal to six months at another to…), it is being used to communicate work for discussion. ”
I would love to see an analysis of the the final IF of manuscripts that are posted in ArXiv and BioArXiv as pre-prints (This is when we need DataHound!). My gut feeling is that they’re not going to be GlamourMag’s, and most are likely going to have IF <5. Since getting space in higher IF journals is so competitive, who is really going to put their genius idea out there for someone else to poach?
Since a lot of these journals are much faster than CNS (~IF 5 journals in my field have 6-8 week turnaround times), I don't see pre-prints being much of a benefits on speed. However, I do see it being a benefit to the authors if they can't seem get past reviewers. Then the authors really should start to question the underlying hypothesis/approach of your manuscript and ask if there is some fundamental flaw that you need to address. Alternatively, if they are pushing a radical hypothesis they should think about how they are trying to communicate their ideas, which may be pissing off the reviewers and shitcanning their reviews.
LikeLike
March 4, 2018 at 7:19 pm
“What I am currently seeing in study section by the way is that people are listing everything and anything in the new biosketch format, from abstracts to preprints to in preparations”
That suggests that what is being cited should be paid attention to, more than what is being mentioned in the description.
Biosketch guideline says:”Note that while you may mention manuscripts that have not yet been accepted for publication as part of your contribution, you may cite only published papers to support each contribution.” Not sure what recent guidelines say with respect to preprints.
LikeLike
March 4, 2018 at 7:25 pm
Oh crap. Just realized you can cite Research products. I think anything can be a Research product.
LikeLike
March 5, 2018 at 9:00 am
@TK: “Oh crap. Just realized you can cite Research products. I think anything can be a Research product.”
The original issue was that they wanted people to be able to list data products (gene entries in databases, software tools). Along with this, it was sold as allowing patents (although you could always have included those) and respecting the different requirements of different fields (in some engineering fields, the conferences are harder to publish in than the journals). In practice, however, everything and anything is being included in the list.
The current biosketch is a total disaster. Anything and everything can be listed in a citation. Furthermore, the citations are separated by “contribution”, which means that people are using the citations to justify the contribution and thus are repeating citations if they contribute to multiple citations. I have seen biosketches that when I read it in depth appear to have 24 papers listed (the maximum allowed 4 + 4 per contribution (5 contributions permitted)) but only have one or two real papers. The funny thing about this is that when I went to pubmed, I found one of these people actually had 7 papers published in real journals.
And, of course, the contribution is about selling yourself. It’s about trying to claim credit for doing something important in a field while not pissing off the reviewers, who might well be in that field.
We have begged NIH to go back to the old biosketch, but they aren’t having any of it, anymore than they will go back to the old scoring system.
LikeLike
March 5, 2018 at 10:46 am
@Philapodia: Standards and practices vary greatly by discipline. The arXiv was initially started as a repository for pre-prints in theoretical and computational physics (high energy, condensed matter, cosmology). I’m not in those fields but I interact with people who are and from them it seems clear that 1) a paper posted to the arXiv is considered as “real” as a paper that appears in a journal, 2) the time stamp on the arXiv posting is accepted as establishing priority, 3) arXiv papers are citable, 4) the most prestigious journal for physics other than Science and Nature is Physical Review Letters, and nearly all theory papers that appear in PRL appear on the arXiv first.
It’s a different world from the biomedical sciences.
LikeLike
March 5, 2018 at 11:27 am
qaz- infinitely worse than a published paper? You ARE trolling me. If I read two papers and find them equivalently good, the seal of peer review is meaningless to me. As is thenJIF. The science is the thing. Once I have engaged, my opinion of the finding is all that matters (perhaps buttressed by expert opinion I solicit directly on parts I don’t understand as well).
LikeLike
March 5, 2018 at 11:30 am
A preprint clearly shows that you have something finished enough that you think is ok to be put out there a
So close and yet so far. The point is that anyone can look and decide for themselves whether the preeps is nearly a paper or just a start.
LikeLike
March 5, 2018 at 11:49 am
“my opinion of the finding is all that matters (perhaps buttressed by expert opinion I solicit directly on parts I don’t understand as well).”
Isn’t that the point of peer-review in the first place, letting people know from the onset that a manuscript been deemed reasonable by subject matter experts? If you’re not an expert, making the determination that the manuscript is “good” for yourself and asking for subject matter experts later on seems to be a hell of a lot less efficient than letting it go through peer review in the first place, where multiple people have vetted it.
This whole concept seems similar to trying to cite a blog post, and seems ripe for cherry-picking data.
LikeLike
March 5, 2018 at 1:07 pm
Agree with DM , the likelihood of getting any comments at all is rather low. But, that still doesn’t negate the use of preprints altogether. I’ve found them to be useful in several ways –
1) We’re trying to publish something we know is “hot”, so there’s a good chance some BSD will sit on it at a glam journal for months before finally rejecting it (and then magically their own paper on the same topic comes out a few weeks later). Submitting a pre-print simultaneously with a regular journal submission (and tooting about it in the cover letter and on the face page of the manuscript!) is a formal way of saying “don’t try any of that scoop shittio buddy because it’s already out there!” So far (N<10) it's been successful – everything we've preprinted has gotten reviewed quickly. It's almost as if the reviewers know somoene's watching.
2) NIH progress reports (RPPR). That shit just has a habit of creeping up on you fast (like 8 months into the funding year kind of fast). Even if the paper is ready to go out the door, a DOI from BioRxiv helps the program officer see that it's real and not just blowing smoke.
3) Along with other open science initiatives (like posting full data sets on figshare), it works well as a bullet point in the rigor & reproducibility section of grant proposals. It's a way of saying "we're so confident in our data we're willing to just put it all out there for eveyone to pick apart". Being able to point to a good set of transparent practices is kinda important for grants these days.
4) Alternative communication is good if people can't say what they want to say in a formal review. In one specific instance, I received a great set of email comments from someone in lieu of a BioRxiv comment, and proceeded to incorporate those comments into the revised version of the paper at a regular journal. Turns out the commenter was one of the reviewers (as admitted later) and just didn't feel comfortable saying what they said via the formal route (this included tearing apart a competitor's work and requesting citation of their own work, both of which were valid points we'd missed). The preprint and back-channel email provided an alternative route for communication, resulting in a stronger paper.
5) The other great use for preprints is they make fantastic candidates for journal clubs. I've also used them as discussion papers for classes, including nominating a class member to post the group's comments afterward. It's empowering for the students to realize their opinions on a paper can be more than just scribbles in the margin of their own copy.
And of course, if'n you don't agree with the BioRxiv comments, you can always just respond with:
LikeLike
March 5, 2018 at 1:08 pm
Dammit, no GIF embeds, WTF?
https://giphy.com/gifs/molly-welker-F3G8ymQkOkbII
LikeLike
March 5, 2018 at 3:32 pm
Philapodia: letting people know from the onset that a manuscript been deemed reasonable by subject matter experts?
What does your experience with peer review tell you? Mine says I am not taking the fact of peer review at some journal as any sort of endorsement of the science. If I can’t understand it, I’m going to ask someone I trust.
LikeLike
March 5, 2018 at 3:41 pm
qaz: The current biosketch is a total disaster.
I disagree. I submit that you only say this because you are one who was perfectly advantaged under the old version and have no relative advantage with the new. There are people who contribute in a way that is not easily reflected on a simple list of papers and it is fair that they are able now to plead their case.
My only problem with it are people who eff up the link to their myNCBI (or PubMed if their name is unique).
qaz: And, of course, the contribution is about selling yourself. It’s about trying to claim credit for doing something important in a field while not pissing off the reviewers,
as has ever been case for the biosketch. in the days of “selected” pubs, those with extensive CVs made sure to list their most Glammy stuff even if it was 20 years old and had nothing to do with the current proposal.
qaz: I have seen biosketches that when I read it in depth appear to have 24 papers listed (the maximum allowed 4 + 4 per contribution (5 contributions permitted)) but only have one or two real papers.
You are are welcome to score the allowable contributions as you like. As is anyone else. Some will view preprints as “real papers” and some will not. Any PI is free to only list “real papers” however they see them and risk the consequences.
LikeLike
March 5, 2018 at 3:42 pm
ThamizhKudimagan: Biosketch guideline says:”Note that while you may mention manuscripts that have not yet been accepted for publication as part of your contribution, you may cite only published papers to support each contribution.” Not sure what recent guidelines say with respect to preprints.
NIH posts notices for a reason, people:
https://grants.nih.gov/grants/guide/notice-files/NOT-OD-17-050.html
https://grants.nih.gov/grants/how-to-apply-application-guide/forms-d/general/g.240-r&r-seniorkey-person-profile-(expanded)-form.htm#Instructions
LikeLike
March 5, 2018 at 8:37 pm
What Ola said.
And, I have had genuinely useful interactions with people who have posted their pre-prints, whom I’ve contacted with a follow-up question about their data. We like to do these for lab meetings or journal clubs too–I think it’s great for trainees to learn to evaluate a manuscript without it having the imprimatur of Neuron or what have you. It’s really good training for reviewing actual manuscripts!
LikeLike