Cell on "a unit of scientific advance"

January 26, 2009

Noah Gray of Nothing’s Shocking pointed to a recent editorial in Cell and whipped up a little analysis of three Nobel laureate’s publications in Nature in response. The Cell editorial (in part) and Noah’s analysis (in the main) focus on the current reality that many GlamourMag articles come with a host of extra supporting material that did not make it into the print article.
Actually, I think Noah may have mistaken the tone of the Cell editorial a trifle. It said:

One issue in particular that we at Cell will be focusing on in 2009 is redefining what constitutes a “publishable unit” in the age of electronic journals and how we can best present the information content of a scientific article online. The vision in our crystal ball is still blurred, but some key elements are beginning to take shape. The scientific article of the future will no longer be tied to the constraints of a printing press and will take advantage of all the opportunities afforded by the web to introduce a hierarchical rather than linear structure, increased graphical representations, and embedded multimedia. Inherent in our thinking about the scientific article of the future is the need to address the current unchecked growth in the amount of supplemental and supporting material and to identify constructive, well-defined guidelines for what is reasonably and appropriately included in a unit of scientific advance.

It will come as no surprise to my readers that I am NotAFan of the Supplemental Online Material which has become an obligatory part of scientific communications in certain journals. I am distinctly old-fashioned and think that when a communication is submitted, reviewed and accepted for publication on the basis of a body of text and figures, well, those should be part of the article. Not as some addendum that you have to go searching for, presented in rough manuscript format. A real part.
I am not so optimistic as to read the Cell editorial as any promise of reform however. If I am not mistaken, of the triumvirate of the GlamourMagz, Cell is the one that still presents something like real length descriptions, is it not? I sort of read it as saying they were going to put even more crap online, but I hope I am wrong. I mean sure, they take a few swipes at the competition, Nature and Science. Perhaps this is going to be the first shot in a little inter-Glamour warfare? As I said, Cell is already on the longer side so perhaps they can best afford a principled cessation of Online Supporting Material. They can then use this new discussion of a “unit of scientific advance” to beat up on Nature and Science for their discordant and hypocritical practices.
Pehaps I am cynical. Perhaps Cell will be leading the charge to roll back this asinine situation we are in under which it takes five years of work from at least a half a dozen doctoral (or in training) scientists to end up with one GlamourMag pub. Which presents maybe a fifth of the potentially useful data and maybe a twentieth or less of the effort involved. All of which gets lost to science because it doesn’t “fit”.
Now although I enjoy the points Noah Gray made, there is one little bit of obvious DM bait included. I should mention, I suppose, that Noah is not only a blogger but an editor at Nature (previously of Nature Neuroscience) for those who haven’t been following along.

As the editors of Cell put it, if the publishable unit is going to change, it is going to take the cooperation of editors, authors and reviewers working together to find a reasonable and appropriate standard for publications. Because, after all, if our reviewers told us that 4 single panel figures were an exciting story that was highly impactful, we would publish it. Alas, they are not giving us this advice.

All the fault of the reviewers, isn’t it? I mean, the editorial staff at Nature are just inert actors in this whole drama right?
It is a cycle Noah. You know this. The “best science” and the ProperWay to construct a journal article is defined by what you publish. So the only people who get in are those that agree, endorse and wholly benefit from the status quo. And, as we’ve discussed before, are considerably motivated to define “best” and “appropriate for Nature” in ways that categorically rule their own labs’ type of work in, and other labs out, to the greatest extent possible. That way, the competitive field is narrowed. Is it any wonder that what is considered worthy of glamour magazine acceptance at any given time marches in lockstep with technology? When viewed with hindsight it is clear time and again that these categorical rules for inclusion of first molecular techniques, then knockout mice, then gene arrays, then… have little to do with the scientific brilliance and more to do with technological inclusion criteria.
So don’t blame it on “the reviewers”, my friend. Accept your own part in the self-referential system. If you agree that Supplemental Material is stupid, bring the editorial hammer. State that the published article should stand on its own, full stop. Refuse to send out manuscripts that contain way more data than will fit your eventual publication. Turn back reviewer requests for same.
Either that or just publish abstract books. Put the whole article online as one piece. Oh, ok, allow one teaser cover-image type figure. It’s not too different from what you do already, right? .

No Responses Yet to “Cell on "a unit of scientific advance"”

  1. Well, I think you do have to put a little of the blame on the reviewers. There is a compulsion on the part of many reviewers to feel like if they don’t demand some additional experimentation, they haven’t done a good job. It is difficult not to succumb to this, and it leads to a scenario where in order to publish a paper in C/N/S you usually needs to combine a vast array of different technical approaches in a single paper.


  2. DrugMonkey Says:

    That’s why I use the term “cycle” holmes. Everybody plays and everybody contributes.
    OTOH, the GlamourMag editors have a unique ability to just put down a foot and make major changes on an instant. What, you think people are going to stop submitting to Nature if they say “no more supplemental nonsense under any circumstances”? pshaw.


  3. whimple Says:

    You don’t have to play. When the reviewers ask for more data, politely say no and explain why. Then let the editor earn his/her salary by making a decision.
    This approach also has the not-to-be-underestimated value of avoiding spending another year trying to please the reviewers and then ultimately failing to do so, or getting scooped, or not getting your grant renewed.


  4. Becca Says:

    Curse those supplementary data! Curse them!
    I mean, there’s nothing worse than actually seeing video of biological phenomenon like sporozoites bursting out of hepatocytes… oh wait, I’m sorry. I meant there’s nothing more “ZOMG INCREDIBLE COOLNESS!1!11!!eleventy!111!” than videos!
    Face it DM, you’re just an old fogey on this one.
    When digital video gets cheap enough they can send me animated dead trees, I’ll get on board about the evils of supplmentary files.
    That said, a lot of the issue I do take with supplementary data could easy be resolved by a simple html coding when I go to “save pdf” / “print” if it prompted me “do you want to add the supplementary data to the file?” / “do you want to print the supplementary data (x pages) too?”
    That would be a distinct improvement. Nothing worse than scanning really cool article, printing it out to read over lunch, and then not having an important figure (note that since I am a grad student ‘boring technical details’ like ‘the timecourse we did to figure out how to do the rest of our experiments’ are really amazingly frequently important figures).


  5. James F Says:

    Sure, supplementary material should not be superfluous (“Look at how many figures I’ve got! The SI is five times the length of the actual paper!”), but length restrictions (Nature letter, Science report, regular PNAS paper, etc.) make it virtually unavoidable. I’ll take it over a suspicious “data not shown” any day. In addition, I believe some journals have (or are moving toward) an option where you download the article and supplementary material as one big pdf (although even this has its limits, e.g., giant Excel tables).


  6. Alex Says:

    There’s a lot to be said for the concept of a long article over a short article. However, there’s also a lot to be said for the concept of a short article. The problem is that we’ve got some journals that were for a long time doing just fine with short articles, and then when it became possible to put materials online they decided to go for everything you’d expect from a long article but maintain the fiction of a short article.
    The one good thing about supplemental materials is what Becca pointed out: You can’t put video on paper. So for stuff that has no real place in dead tree format, I think supplemental materials make perfect sense. But using supplemental materials to go for more length in dead tree format is not the way to go.
    In physics, the way these things usually work is that the most important results go to Physical Review Letters, where articles are no more than 4 pages. The articles are self-contained, but generally around the same time (and generally shortly after) a companion article comes out in another Physical Review journal, a much longer article examining everything in more detail. Sure, there are some PRL articles that don’t need a companion article (in theoretical physics, a mathematical result may be significant but simply not need a 12 page companion article), and they just do a PRL. But generally, if you have a lot of data on a system, you write a 4 page PRL, describe the basics of the method, describe the most significant results, and then write a longer article for one of the companion Physical Review journals. This is probably fairer to authors, because while they might spend just as much time as they’d spend dealing with endless reviewer requests for supplementals, they get two papers for their troubles, and one of those papers might actually come out in a timely manner.


  7. DrugMonkey Says:

    James, you are missing the whole point. Why are there these asinine length restrictions if 75% of the published work requires “extra” figures? Something is wrong with this picture.
    becca, I’ll take your point for any non-traditional media/video stuff that cannot be represented in print. But that is a vanishingly small part of the picture and you know it. if every downloadable version of the manuscript included the missing data I would be partially mollified…but then why not just publish abstracts and be done with it. And glomping unrefined manuscript onto typeset article is just….crass.


  8. Alex Says:

    I should amend that a bit: Supplemental materials on the nitty-gritty of methods don’t bug me. It’s better for the scientific community that stuff like that be shared to enable replication, testing, and building on previous results. But when results essential to the conclusions drawn are put in supplementals, that bugs me. The occasional “data not shown” is OK if they basically say “Tests on all of the other samples gave graphs with the same slope within 1% [data not shown]” but something like “We also ruled out [major competing explanation] using blah blah (see supplementals)” really bugs me.


  9. Dave Says:

    Ah, whimple, but therein lies the problem. Back before supplemental data, authors could always politely decline some reviewer’s absurd request as being ‘outside the scope of the present study’, and editors would have to accept this excuse as reasonable due to article length constraints. But nowadays there are effectively no article length constraints so authors have no recourse along those lines. I agree with DM that editors have to draw the line.
    In the mean time, I agree that reviewers (that’s us, remember) need to stop asking for experiments merely to ask for experiments. I too hate supplemental data. The only time as reviewer I have ever asked for something that I thought might be supplemental data instead of a critical replacement for an already-included figure was recently when the authors kept referring to important ‘data not shown’. I said that the ‘data not shown’ mantra was unnecessary and inappropriate; if the data exist and are reliable they should stick it in a supplemental figure.
    As for Cell: There is a need for high-profile journals that publish longer papers, and Cell Press provides an important venue in this regard. Maybe their new editorial policy is a sign that they’re shifting from favoring large-scale descriptive science to stuff that actually has a hypothesis and conclusion. That would be good. Cell has gotten kind of crappy lately.


  10. DrugMonkey Says:

    “data not shown” is basically equivalent to “we think so but we didn’t actually bother to verify it”. or “our mother’s eye found that in the data but we know if we showed it you would laugh us out of town”.


  11. James F Says:

    James, you are missing the whole point. Why are there these asinine length restrictions if 75% of the published work requires “extra” figures? Something is wrong with this picture.

    I contend that, compared to a decade or so ago, high-throughput and systems approaches (apologies for using both of those terms) translate to more data required to more a paper competitive at Nature, Science, and so forth, to “make the cut.” Would it be better to increase the thickness of Science and Nature to that of JBC? I’m not being a wiseacre, it just seems like a choice between a very large or a very concise weekly journal. The latter, of course, is more amenable to flashy, bite-size reports.


  12. James F Says:

    edit: “…to make a paper competitive…”


  13. MBench Says:

    DM, I agree. I also think this is evident in the explosion of actual authors on a typical GlamourMag publication. Yes, if you and 100 of your closest friends sequenced a genome, put all your names there. But if you discover something truly new and interesting to a broad scientific audience, do you need fifty extra assays and techniques applied to look at it from every possible angle? With twenty other names on the paper as a result? Better to have one punchy, short GlamourMag publication and let the other experiments form the nuclei of several other papers. This ends up benefiting everyone, the authors (more first authorship to go around, perhaps fewer dual-first-authors?), the PI (more publications and more attention to each smaller project), and the journal readers who get distilled science in the GlamourMagz that is fresher and more direct and to-the-point…


  14. JD Says:

    [“data not shown” is basically equivalent to “we think so but we didn’t actually bother to verify it”. or “our mother’s eye found that in the data but we know if we showed it you would laugh us out of town”.]
    Or it could be that the reviewers asked for 15 different sensitivity analyses and it just got repetitive/boring to have table after table with the same number to 2 decimal places.
    So saying “The result stands if we restrict the population to alpha centurians, restrict based on past gamma ray exposure, adjust for radioactive spiders as both spider and radiation or stratify the by the other reviewer pet variables (data not shown).” could just be a way of verifying:
    yes, we checked this strange hypothesis. No, it didn’t matter. But yes, it would be shorter to publish the data set than then number of follow-up analyses requested.
    Not that I’ve experienced this . . .


  15. Alex Says:

    When I solve an equation for a bunch of different parameter values, I don’t plot every single case. I plot the important ones. I might say “Increasing the value of this parameter past X value up to Y value did not change the blah blah by more than 1% past the results shown here.”


  16. acmegirl Says:

    “I am distinctly old-fashioned and think that when a communication is submitted, reviewed and accepted for publication on the basis of a body of text and figures, well, those should be part of the article. Not as some addendum that you have to go searching for, presented in rough manuscript format. A real part.”
    DM, this is by far the thing that annoys me the most about Supplemental Materials – if I can’t understand how the authors came to their conclusion, and therefore have to print out the Supp, why in the world can’t they have formatted it in some way that is readable and doesn’t take up so much paper? The digital revolution has come and gone in publishing – it’s not too much work to create a template and plug the material into it (or even ask the authors to do it themselves), rather than forcing us to read something that looks like a high school term paper. Damn, some intern could probably whip that up for them. It’s just lazy, as far as I’m concerned.


  17. qaz Says:

    Becca #4 – Amen! Why can’t supplemental PDF be included with the online version of the paper? (Ok, video can’t. But none of us are upset about the ZOMG video in supplemental online. We’re all ticked about the supplemental PDF that didn’t fit in the main paper.)
    I have spoken to several of the editors of these journals at conferences and suggested that they automatically fold the supplementary data into the online version of the paper, and they always say “that’s a good idea” but then never do it. What that says to me is that the editors don’t feel that supplementary data is really part of the paper. They feel it’s supplementary. Science doesn’t even edit their supplementary data. You can still find supplementary data that says “DRAFT” on it! Nature Neuroscience still makes you click every goddamn figure separately for supplementary data.
    What ticks me off these days is that supplemental material is bleeding down to the real journals from the glamor mags. I’ve seen supplemental material in JNeurosci! There are no length limits in JNeurosci! Put it in the damn paper.
    Of course, I know why it’s in the supplemental. It’s because reviewers don’t always notice that there is supplemental. (Several journals – JNeurosci and NatureNeurosci, I’m talking to you – make it hard to find the supplemental when reviewing.) So the supplemental often ends up being a place where one finds the B-side stuff that didn’t make the final cut.
    Maybe, as reviewers, we should start trashing papers that put too much in the supplemental material.


  18. anon Says:

    I once had a two page communication that had 70 pages of supplemental. It was awesome. We even had to put some arguments about the mechanism into the supplemental due to the 2 page length restriction.
    I’m proud of that record in a perverse sort of way. I don’t know of a single good chemistry journal that doesn’t have supplemental these days. Due to length restrictions on very prestigious 2-3 page communications in the best journals (they are way more prestigious than long articles because you did something awesome that requires a short and quick communication to show how awesome and hardcore you are), the supplementals get a little out of hand since you’re forced to put part of your argument into there, kinetics, etc…
    It’s now not only the actual experimental part which would never fit into a 2 page communication, where you only present the results and talk about how awesome they are without any backup. The reviewers now treat it as part of the paper too since that’s where a lot of the real action is, and they argue about supplemental figure S5 and the small discussion I had in there about the mechanism.
    Just you wait bio types. It’s going to get a lot worse.


  19. I loathe supplemental material (aside from videos and gene sequences and what have you). I am particularly appalled by supplemental methods. We all know that the devil is in the details with these things–how many times have you said, “Oh, they were using 2.5 mM Ca??” or equivalent, and then revised your opinion of a figure’s impact?
    I’ve been in too many journal clubs where nobody can figure out what the precise methods were for a given figure because the supplemental methods haven’t been treated seriously….and no one but the presenter has printed them anyhow.
    Plus there’s the posterity issue. You think in 30 years when someone goes back to cite your paper, they’ll still be able to read PDFs from 1999? How many file types from 1979 can your computer read now? Since the supplemental is usually offered only as PDF, this could be a real problem. Not to mention what happens when Cell goes belly-up and stops hosting their servers….


  20. Alex Says:

    A few things have lasted since 1979. ASCII is still around. .pdf is already 16 years old and such a popular archival format that I suspect it will last, or at least converters will be around. I mean, you can still buy a tape player!


  21. AM Says:

    I think the current maximum length set for articles have been estabilished due to the space constraints given by the dead tree format. Just ditch the dead tree format already (who reads those anyway?), and make the maximum length reasonable, so that a complete story can fit in. Then you will need no supplemental material. And you will not have the ridiculous but alas increasingly common situation whereby you have “full methods” in the supplementary data because they cannot fit in the paper proper.
    In addition, doing so will allow the magz to save money – and will save a lot of dead trees. Of course, without the print edition, it will be increasingly more difficult to justify the outrageous charges the publishing houses exact for their magz…not that they will care anyway.


  22. BiophysicsMonkey Says:

    “if I can’t understand how the authors came to their conclusion, and therefore have to print out the Supp…”
    This is the key for me. If I can follow how the authors reached their conclusion and why it’s reasonable from the main text alone, then if they want to include their 50 replicates or fancy video or whatever as supplementary data that’s fine.
    Recently for a journal club a student sent us all a paper from Molecular Cell, and it was not remotely clear that the authors had even come close to proving their hypothesis. Everyone arrived at the club meeting puzzled and it was only after the student spent most of the presentation going over supplementary data that things made sense.
    That’s a problem. The editors should catch things like that.


  23. whimple Says:

    You think in 30 years when someone goes back to cite your paper, they’ll still be able to read PDFs from 1999?
    This is an excellent point. I have already run into this… critical supplemental data, that once upon a time was hosted on a server, but then something happened, and it went away, and now it’s not worth the authors’ bother to retrieve it for me.


  24. DrugMonkey Says:

    As it happens, there is an excellent post on the role of academic citation up at Green Gabbro today.


  25. Dave Says:

    DM: The citation discussion was in the other thread.


  26. bsci Says:

    I think DM’s comment that these are becoming printed abstract books with online articles is spot on. I’m not sure that’s a bad thing, if these journals embrace that designation. The printed version becomes more like a new magazine where you get just enough of the studies to understand the importance and basic assumptions of the research. The actual research article is online and shouldn’t be called “Supplementary material”
    I think the key actual change to get there is to make the supplementary material more of an article that can stand in it’s own merit and contains professional formatting and the full narrative flow. A pubmed search would pull up the actual article and not the printed abstract.


  27. Becca Says:

    DM- the reason I said I wanted an html code popup prompt asking if I want the supplementary data is that I’ve seen supplementary data that includes results like microarray data. I love to pour over it to see if My Favorite Gene was upregulated/downregulated when they chumple the quittleblurks. But I am not anti-tree enough to want to print out 72 pages of that nonsense!
    Arguably, all of this sort of stuff should go up on webpages associated with the manuscripts (which would be really nice if they were integrated hyperlink style within the pdf), rather than supplementary figures. But not every lab is savvy enough to set that up for every dataset. Though it’s properly the job of the journals to host it. I feel like that kind of data should be widely available whenever possible. It’s not pretty data (just boring tables), but it’s not the raw crude unrefined sequencing curves either.
    And maybe I’m much more of a cell biologist than you, DM. A fairly large percentage of the supplementary material in the papers I read is video format.
    Side rant: if the journals want to justify their ridiculous prices, what they *should* be providing in is useful digital archiving.
    qaz#17- I wonder, is paper really still so expensive that editors still need to be obsessive over space in the main journal article? Or are they just hoping nobody will judge them as terminally sloppy lazy bums for permitting ‘supplementary figures’ that are formatted in utterly hideous fashions?
    “DRAFT”? Unedited? Only accessible as individual pictures? This is just really poor form on the journals parts.


  28. drugmonkey Says:

    Found this old thread, thinking about the recent LPU discussion. Interestingly JNeuro has decided to cut way back on Supplemental Material recently.


  29. physioprof Says:

    They didn’t “cut way back”. They have banned it completely.


  30. DrugMonkey Says:

    oh right. I had that part about “point to your own damn website” confused in my mind with rare exceptions for some reason. I haven’t paid attention to Cell though. Have they “checked” the growth of supplemental materials? Have they substantially changed the nature of one of their articles since posting that editorial?


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: