Yay! J. Neuroscience Agrees with Me that "Supplementary Materials" is BS and Ruining Science!
August 11, 2010
Beginning November 1, 2010, The Journal of Neuroscience will no longer allow authors to include supplemental material when they submit new manuscripts and will no longer host supplemental material on its web site for those articles.
HAHAHAAHAHA!!! Yes!!!!
whew. calm down, DM, calm down. why are they doing it?
Although The Journal, like most journals, currently peer reviews supplemental material, the depth of that review is questionable. Most well qualified reviewers are overburdened with requests to review manuscripts, and many feel that it is too much to ask them to also evaluate supplemental material that can be as extensive as the article itself. It is obvious to editors that most reviewers put far less effort (often no effort) into examining supplemental material. Nevertheless, we certify the supplemental material as having passed peer review.
True, true. A concern to be sure. [stay calm, DM, stay calm…]
Another troubling problem associated with supplemental material is that it encourages excessive demands from reviewers. Increasingly, reviewers insist that authors add further analyses or experiments “in the supplemental material.” These additions are invariably subordinate or tangential, but they represent real work for authors and they delay publication. Such requests can be an unjustified burden on authors. In principle, editors can overrule these requests, but this represents additional work for the editors, who may fail to adequately referee this aspect of the review.
Reviewer demands in turn have encouraged authors to respond in a supplemental material arms race. Many authors feel that reviewers have become so demanding they cannot afford to pass up the opportunity to insert any supplemental material that might help immunize them against reviewers’ concerns.
w00000t!!!!1111!!!!ELEVEN!!!! YAYAYAYAYAYAY!!!! Damn tootin’!!!!!
Supplemental material also undermines the concept of a self-contained research report by providing a place for critical material to get lost. Methods that are essential for replicating the experiments, analyses that are central to validating the results, and awkward observations are increasingly being relegated to supplemental material. Such material is not supplemental and belongs in the body of the article, but authors can be tempted (or, with some journals, encouraged) to place essential article components in the supplemental material.
OMGWTFRUKidding me?????!!!??? YES!!!! YESSSSSSS!!!!!!!!!!!!!!! Exactamudo correcto!!! YES!!!!!!!!!!!!!!
Three cheers for the de-GlamourMagification of the Journal of Neuroscience!!
[h/t: Dr. Jekyll & Mrs. Hyde for alerting me to this important news]
DrugMonkey 
August 11, 2010 at 2:30 pm
SRSLY. Eighteen thumbs up for Maunsell and whoever else was in on this.
LikeLike
August 11, 2010 at 2:30 pm
I agree with you too: http://bit.ly/a0kMFn
Its a glorious day!
LikeLike
August 11, 2010 at 2:32 pm
Yes!!!!!!!!!!1
LikeLike
August 11, 2010 at 2:33 pm
Does this mean I can start submitting here without fear of being slow-roasted over coals for 10 extra figures?! hurrah!
LikeLike
August 11, 2010 at 3:13 pm
What about things like videos? Are those to still be included as supplemental materials? (I don’t know how often J Neurosci manuscripts include videos, but in my field, I actually wish *more* papers included supplemental videos)
LikeLike
August 11, 2010 at 3:16 pm
Which also helps deal with the problem that normally the supplemental data don’t get included in pdfs etc. It’s a good solution.
LikeLike
August 11, 2010 at 3:22 pm
That’s some good news! I’m glad somebody had the gumption to do it.
LikeLike
August 11, 2010 at 3:54 pm
It is about fucking time! Let’s hope some other journals follow suite. Now, I just have to figure out how I can try to submit something to J. Neuroscience.
wOOt!!!!!111!!!!!1!!1!1!111111
LikeLike
August 11, 2010 at 4:03 pm
HOORAY!!! It is about fucking time. I hope that other journals jump on that bandwagon-now I just have to start doing research that I can publish in J. Neuroscience.
LikeLike
August 11, 2010 at 4:04 pm
Praise Jeebus!
LikeLike
August 11, 2010 at 4:39 pm
Jeezus fucke, holmes. You’re splooging all over the fucken blogge. Clean that shitte uppe.
LikeLike
August 11, 2010 at 4:57 pm
J. Neuroscience FTW!
All of the reasons they list are true for ecology journals. Sometimes the supp mats are longer than the paper. I have heard horror stories of reviewers demanding all kinds of stuff be done and put in the supp mats for no valid reason.
Another problem with them is that they are usually (always?) behind the journal’s paywall. Yet they are supposed to be part of the article. Doesn’t work that way.
I can think of some things that do need to be in an online, rather than printed, form. Videos. Large data sets (and NSF is about to implement a policy that every research proposal must include a Data Management Plan on how all data will be made available to everyone). Computer code. Maybe a few others.
LikeLike
August 11, 2010 at 5:29 pm
YAYAYAYAYAAAAYY!!!!!!! WOOOOOT!!!!!!!!!!!!!!!!!!!
LikeLike
August 11, 2010 at 5:50 pm
[…] This post was mentioned on Twitter by Drug Monkey, Justin Kiggins, mavericksmusing, Virginia Hughes, Martin Striz and others. Martin Striz said: “Supplementary Materials” is BS and Ruining Science http://j.mp/bWOAkl […]
LikeLike
August 11, 2010 at 8:10 pm
This is great from the “protecting the quality of the scientific record” perspective, but what about the consumption side of it? I for one like the idea of being able to read papers at different levels of detail. If I’m reading in an area that’s not my bread and butter work I don’t need all the technical stuff, but I may want more than the abstract or just the figures.
LikeLike
August 12, 2010 at 4:28 am
It’s a real problem for the journal to host all this extra stuff. They have to rent their cyberstorage and I believe there are charges for people accessing this stuff. So I can see another reason for J Neurosci to junk supplementary info. Authors can still host their own data somewhere else if they want to make files available for readers perusal (and they can put that in the Data Management section of their NIH grant).
LikeLike
August 12, 2010 at 5:43 am
Let’s hope that other journals take the cue.
LikeLike
August 12, 2010 at 6:34 am
WOOHOO. Now all they have to do is convince the reviewers. Will there be a BIG HONKIN SIGN saying “NO you will NOT demand fifty fraktillion extra experiments to be put in the supplemental materials”?
LikeLike
August 12, 2010 at 6:43 am
Mizumi, that is why there are review articles and textbooks. But really, you can’t skip your way through a paper, ignoring the controls or methods if it doesn’t interest you? Is this such a burden?
LikeLike
August 12, 2010 at 7:03 am
Yay! So when do you think Science and Nature will see the light and get rid of their system of pointlessly harsh article length restrictions followed by textbook of supplementary materials per article? I’m waiting. waiting…
I will note, as Jason Goldman said, that there are some things that seem ideal for supplemental materials. There are mostly large objects that enrich our understanding of the information in the article. This includes movies, which might be too large to embed in every pdf in the short-term future. I’d also add information slightly closer to the raw data. For example, if a paper has a chart summarizing the averages across a population or multiple recordings, it’s sometimes useful to glance at the subject/site specific data that underlies those averages. Taken to a very unrealistic (for many reasons) extreme, it would be cool if the journals become hosts for the raw data behind the studies they publish.
I strongly, strongly agree that the methods section must be self-contained in the main manuscript and supplemental material shouldn’t contain distinct experiments or analyses.
LikeLike
August 12, 2010 at 7:12 am
In the golden era of papers that reported ‘new genome X’, the methods on how the authors produced that data were one of the most important parts of the paper. These days they are almost invariably shunted off to the sidings that is supplementary material.
This news pleases me. As others have pointed out, journals such as Nature and Science seem to be abusing the role of supplementary materials. I reviewed a recent Science paper for our lab two weeks ago. The printed paper was four pages. The supplementary material ran to 53 pages.
I’d be happy with a rule that simply stated that the length of supplementary material cannot exceed the length of the paper.
LikeLike
August 12, 2010 at 7:35 am
I seriously hope that not all journals go this route, and I’ll tell you why . As part of my postdoc, we did an MLST project on a common pathogen in the food industry. We managed to get strains from all over the world, from state and federal agencies, and from international organizations. In the end, the project had several hundred isolates. Where were we supposed to list them and all the supporting data (date collected, place collected, food it was isolated from, serotype, etc etc)? In the article itself? It would have taken a dozen or so pages just for that one table! And that’s not including the several phylogenetic trees we generated all of which took up an entire page. It would have been half the issue if we had incorporated it all into the text. It seemed that that was a PERFECT use of the supplementary material section. Instead of the editors taking proper control of their reviewers and authors, they’ve banished the practice altogether … bye bye baby, but at least we don’t have to worry about that bathwater any longer.
LikeLike
August 12, 2010 at 9:15 am
I agree with most of these points, but if some journals are convinced there is a need for Supplemental Material, at least have the courtesy of providing a combined pdf with all data together (I think Cell does this now) and provide Methods in the fashion of some of the Nature journals where online only methods are part of the paper as a whole, and integrated into the pdf. If they aren’t going to get rid of it, do it right. However, in many older papers there were things like “data not shown” and being able to show these (many times) controls is nice. If reviewers didn’t abuse the notion, and editors weren’t so mindless or greedy, they’d prevent the inevitable supplemental data creep that is such an abomination.
LikeLike
August 12, 2010 at 9:21 am
If one can cross-post blog posts, why not cross-post comments? 🙂
Maybe there should be several papers. Or perhaps a journal should be selected that doesn’t put artificially constraining length requirements on the table.
This is a tough call. Usually you are trying to reach a particular readership. While I really support this move on the part of J. Neurosci, are they not just pushing the burden of storing large files onto other servers, and not guaranteeing the perennity of said extraneous information?
If they really enforce the self-contained nature of the articles, then my point is moot. Primary data from -omics studies can often be uploaded to community-recognized, permanent resources (none of us are afraid to lose PubMed, right? then no fear for GEO). And just the relevant charts included.
I’ve published quite a few papers with appendices/supplementary material. But I do resent having data that I consider good get relegated there based precisely on that back-and-forth the editorial describes. Relegated, because as Eric Lund states (on the other site), human nature is not to look at it unless there is an unusually vested interest.
LikeLike
August 12, 2010 at 9:25 am
Agree with all the points above and add 1 more — supplemental references (and the related problem of artificial, journal-imposed restrictions on number of references permitted). If an article is important to the design or interpretation of your study, it should be in the main reference list (regardless of length) and “count” towards the citation history of that article. If not, chuck it. Online-only supplements have the potential to distort the citation history of the literature, especially with respect to methods-development articles.
LikeLike
August 12, 2010 at 11:30 am
I had a discussion with a colleague recently, who was told by the reviewers to cut his discussion section down to the stated limits of the journal. His “solution” was to submit a “supplementary discussion.” I told him that would not (and should not) fly. He tried anyways, and (thankfully) he failed to convince the editors to give him extra space for his beautiful ideas. How could this highly intelligent person have such a tin ear on this issue? Hubris, I guess.
LikeLike
August 12, 2010 at 12:01 pm
Beaker,
Many journals allow supplemental discussion in various forms. The annoying part is sometimes this is where controversial issues are stuffed like “actually, this previous paper published in the literature is total bullshit. We’re diplomatic in the main text, but we’ll spell out our problems here, below the fold as it were.”
Oy.
LikeLike
August 12, 2010 at 1:24 pm
NC has a really good point. As far as I know supplementary refs are not tracked in Scopus or ISI. In the H-index (or whatever metric) world we live in now that is a major issue.
LikeLike
August 12, 2010 at 1:55 pm
Agreed with N-C and juniorprof regarding supplemental citations being a wretched distortion of reality. Recently I actually tried taking three Nature papers I was reading and comparing whether the citations in the supplemental material were preferentially tilted towards lower-tier journals (for methods development type reasons as suggested). It gave me a headache and there were too many journals I didn’t always recognize and there weren’t really enough refs to get significant data anyhow, but I bet someone with time on their hands could do an interesting study of that.
LikeLike
August 12, 2010 at 4:15 pm
I’ll have what he’s having.
LikeLike
August 13, 2010 at 6:35 am
I understand what you’re all saying: integral figures and integral methods should not be relegated to supporting information. On the other hand, the Supporting Information is exactly the right place for complete experimental information and characterization for future researcher. In better journals in organic chemistry, typically the complete details of synthetic protocol and complete characterization data (including tabulated data and also primary 1H and 13C NMR data to verify identity and purity) go in the S.I. These descriptions and data, while of no value to the general readership (and thus not appropriate on the printed page), are of exceptional value for anyone repeating the work or doing related work – literally they give exact, detailed instructions for synthesis, purification, and characterization of the compounds. They also endow the work with a stamp of reliability and generally assure that the work can be repeated easily. The disappearance of S.I. would be a travesty in my field. Indeed, some journals (cough, Elsevier, cough) do not require S.I. and the work in these journals is broadly considered much less reliable and repeatable (mainly because you’re guessing at the exact protocols you should use throughout).
One of the biggest challenges of working in biology is that methods sections are woefully incomplete, so literally even things as trivial as expression of a new protein involve wasting researcher time optimizing something that somebody else has previously optimized. For less trivial tasks, the problem is even more severe. Repeatability of work should not rely on knowing or learning special tricks developed by the original scientist. Full, complete, detailed methods sections in the Supporting Information would immensely enhance the quality of the science and should be a basic condition of publication.
LikeLike
August 15, 2010 at 3:47 am
I hope the glamor mags don’t follow suit. The fact that
A) all the actual experimental information in glamor mags is in the supplement, and
B) The supplementary material is not pay walled
means that I can actually keep up with glamor mag science by reading the abstract + supplement, and drawing the obvious conclusions. There is no need to shell out $$$ for the actual article.
LikeLike
August 23, 2010 at 12:50 pm
[…] To some extent, this attitude has changed. The technological means to share data have greatly expanded and it is not uncommon to see a large amount of data and subexperiments buried under the heading of “Supplemental Materials” in online publications. Indeed, this may offset the problem caused by non-reproducible studies slightly, but the habit of attaching more data to publications has brought its own problems: an intellectual arms race between submitters and reviewers has caused the supplemental section to grow ever more lengthy, which recently caused the Journal of Neuroscience to abolish them. […]
LikeLike
October 1, 2011 at 9:44 pm
I have observed that online education is getting popular because obtaining your degree online has developed into a popular selection for many people. A lot of people have not really had a possibility to attend a regular college or university however seek the improved earning possibilities and a better job that a Bachelor Degree offers. Still some others might have a college degree in one training but would want to pursue another thing they now have an interest in.
LikeLike
April 15, 2012 at 6:52 pm
I simply wanted to thank you yet again for the amazing blog you have designed here. It is full of useful tips for those who are actually interested in this specific subject, specifically this very post. Youre really all actually sweet along with thoughtful of others and also reading the blog posts is a wonderful delight in my opinion. And exactly what a generous gift! Ben and I usually have fun making use of your ideas in what we have to do in a months time. Our collection of ideas is a distance long and simply put tips might be put to good use. Rent a bedroom
LikeLike
July 13, 2012 at 6:57 am
Good day! I simply want to give an enormous thumbs up for the great information you’ve right here on this post. I will probably be coming back to your weblog soon.
LikeLike
February 24, 2013 at 9:47 am
Forget diet pills, porn or viagra, our hcg injections are the best for your health
LikeLike
August 22, 2013 at 10:54 am
You ought to be a part of a contest for one of the best websites online. I will recommend this site!
LikeLike
November 19, 2013 at 3:19 pm
[…] Drugmonkey: Yay! J. Neuroscience Agrees with Me that “Supplementary Materials†is BS and Ruining Scien… […]
LikeLike