Putting Impact Factor Restrictions on Journal Club Articles is Stupid and Unscholarly

May 4, 2011

In the “ya learn a new one every day” files comes a Tweet today which says:

We are required to pick J Club articles from high IF journals.

This is stupid. Just…..stupid. And unscholarly. The notion that a laboratory uses this as a criterion for which articles that are permissible to discuss is just….foreign to me.

Journal club, for the uninitiated few in my audience, represents a meeting in which the members of the laboratory discuss a scientific paper. Methods can vary but usually the paper is selected and sent to the laboratory staff about a week in advance of the meeting. One of the lab members generally opens with a presentation of the paper. Maybe at the power-point level with a couple of figures copied in, perhaps with a lot of background context involved as introductory material…but perhaps not.

The reasons for selecting papers vary tremendously in purpose and intent.Just a few I can think of off the top of my head.

  • A review of a foundational paper in the field- for general didactic reasons (“kids these days never read the old stuff“) or because new developments warrant a reconsideration of that prior paper.
  • The very hottest and newest paper in the field is going to revolutionize everything!!!1111!1
  • A paper (foundational, brand new or otherwise) from a tangential field that the laboratory needs to be moving into.
  • Techniques that are of interest to the lab’s current or future directions.
  • A super-complicated paper which is hard to understand and the efforts of the lab brain trust can bring clarity.
  • Education of the most junior trainees on how to properly read and assess a paper for meaning, breadth, scope and clarity.

I don’t see where these goals are served by using some sort of IF cutoff for the journal in which the paper was published.

As we are all well aware, the actual scientific impact of a given paper on the ongoing work of a specific laboratory is entirely uncorrelated with the IF of the journal in question. Unless, of course, your laboratory exists solely to butt-sniff other GlamourMag laboratories and/or solely to publish in the GlamourMags by whatever means necessary. If you are interested in actual science, however, it is inconceivable to me that a lab which can sustain a journal club (i.e., greater than two members) can never have any interest, in or gain any value from reviewing, papers from a variety of journals.

There are downstream implications of this. Another key feature of journal club, especially as the groups get larger, is that a given person will notice a paper that would be of general interest that the other members of the lab (ok, the PI) haven’t noticed yet. And an important function of Journal Club is this literature-scouring part. If the lab head inculcates the group to only focus on that limited subset of GlamourMags (which is what is being done by insisting this is the selection criterion for Journal Club discussions) then those lab members are going to stop reading anything else. They are going to adopt the a smug superior pose that if it ain’t in Science, it ain’t worth reading. This will leave them with an inferior understanding of their own science, first, so this is a big mentorship failure right there. Second, they are going to be unable to serve that literature-filtering and literature-discovering function which, in my view, helps everyone in the laboratory. Most especially the PI, I will admit. If everyone is reading the same four journals there is no hive mind advantage when it comes to covering the scope of possibly-related scientific discovery.

This IF-exclusionary approach will also lead to a follower mentality in the laboratory’s scientific directions. Interesting and novel stuff in science results all the frigging time from a translation of techniques and approaches in one area of science to another. Often times it is the cross-application that turns the ho-hum from one field into something really amazing in another. Interesting stuff also results from long-forgotten minor observations being rescued and followed up with current capabilities, techniques or knowledge that has been developed in the mean time. Someone has to be reading off the beaten path and make the connections to other work for this to occur.

No Responses Yet to “Putting Impact Factor Restrictions on Journal Club Articles is Stupid and Unscholarly”

  1. becca Says:

    I know of lab-run journal clubs, and department or grad program run journal clubs. The later are often required as a for-credit course for the junior students, and are the only place I’ve seen IF requirements. Even where I’ve seen them, it’s aways been something like “pick a paper from C/N/S or *insert list of 3-4 top field-specific journals*, OR come speak with me about a specific exception”

    The principle is to discuss fairly substantial bodies of work (there has to be enough to fill the hour) and actually part of the purpose may be to *inoculate* against glam-journal worship. Many students come in believing far too much of what goes into a printed paper. If the purpose of the journal club is to rip things to shreds, there is something to be said for ripping apart the perceived top science so as to force people to confront the fact you have to give a close reading to a paper to really evaluate the work (and even then, a lot of times you just don’t have enough information to tell if some conclusions are strongly supported or if a limitation of the model is going to screw it all up).

    Like

  2. DrugMonkey Says:

    I can fill several hours on the implications of a single figure in an obscure publication.

    Like

  3. Namnezia Says:

    “usually the paper is selected and sent to the laboratory staff about a week in advance of the meeting.”

    Hahahaha! Usually it’s more like 2 hours before the JC.

    But in any case, maybe this requirement for large departmental JCs is for quality control. While there are many crappy papers in high IF journals, there is often a higher percentage of them in low IF journals. A high IF journal will also likely yield papers of interest to a greater departmental audience. As far as lab-specific JCs, I completely agree with you.

    Like

  4. becca Says:

    I’m shocked.

    Like

  5. studyzone Says:

    When my grad PI started up the lab journal club, one of the first papers we read was from a GlamourMag. Most of us (early-stage grad students and lab techs) focused on/struggled to understand the implications of the figures, and the experiments that generated the data, but I know we all thought it must be a good paper because of where it was published. During journal club, our PI lead us through a discussion that picked apart a key experiment in the paper. With a few specific questions from the PI, we became aware of a glaring inconsistency in the experiment, especially with the data presented in one figure. We had glossed over it, but once we were aware of it, it was obvious. We found out that the paper was later retracted because of a mistake in how a particular data set was analyzed. It was an important lesson for us – don’t judge a paper by the journal in which it is published.

    Like

  6. DrugMonkey Says:

    Somehow I do not get the impression that the triggering comment came from someone who was instructed to present high IF papers only to show noobs that the Emperor is not actually wearing any clothes.

    but I agree that this is valuable. I didn’t suggest that high IF papers should be ruled out….

    Like

  7. BugDoc Says:

    For our departmental JC, we have a requirement for reasons that you stated, Namnezia. The actual requirement was not that the students had to pick a paper from a high IF journal, but rather they had to choose a paper that they considered had high impact from a scientific perspective (could easily be from a field journal). The reason we decided to impose this requirement was that some people would pick papers that were so incremental and narrowly focused that it wasn’t worth reading the paper, even in the presenter’s estimation. Someone actually said this after their own presentation. Without this requirement, I think JC attendance would eventually drop off pretty dramatically….

    Like

  8. Alex Says:

    I sometimes just browse through a journal and pick a random paper that is only tangentially related to my research interests, just so my research students can see the process I use to decipher things outside my background, and the sorts of analogies and connections I draw on to try to understand it.

    Also, not quite a journal club, but a professor at another school once told me that when she’s teaching undergrads how to read journals, she deliberately does not even pick the paper until she’s in class. She wants to be in the same mode that her students will be in when they will start picking papers to read for their assignments. She opens up website of a decent journal with a title related to the subject of the class (e.g. “Journal of large-eared mammals” for a class on Bunny Hopping), randomly picks something with a title that sounds vaguely related to the class, and then reads the abstract aloud to them while interjecting her guesses or comments on what it probably means, or what might be in the rest of the paper.

    I’ve actually found this effective, though I’ve only done it a few times.

    Like


  9. Perhaps the true reason is that they want to give the students practice at trudging through data and methods from the typically poorly explained, yet integral, supplementary material…

    Like

  10. drugmonkey Says:

    I am still waiting for someone to explain how a paper is “not worth reading” because of limited scope. (leaving aside personal scientific interest- plenty of papers that have nothing for us but are chockablock full)

    It may not be worthy of journal club but this statement implies something more universal.

    Like


  11. For our lab’s journal club discussion, we each pick a paper that is pertinent to our project or to our field and give a short presentation (~5min) on why you should read it. So no real IF restriction, just pick a good paper please.

    Like

  12. leigh Says:

    what i’ve found personally is this. people who insist on skimming exclusively from these types of journals also take tremendous pride in showing what theoretically-better scientists/experiment designers/data analyzers they are. and by that i mean, they tend to find a big-shot lab’s glamour publication and voraciously tear it apart. as i’ve seen, this focuses predominantly on the negative and not in a constructive “they could have done this more thoroughly by” or “how cool would it have been if they did” manner that might be of some kind of benefit to the group participants.

    what that says about the journal club group… i have my opinions but i’ll leave them out of the comment.

    Like

  13. Dr. O Says:

    We were forbidden to use N/S/PNAS articles since they often lacked sufficient detail. We had no IF requirement either, only that we get approval if it was not from a (very long) pre-approved list of journals.

    Additionally, my grad PI would always ask her journal club students “have you read it?” before approving articles. If so, and it was a dud, some sort of teasing ensued for the remainder of the semester.

    Like


  14. Dude, you’re such a fucken crybaby. WAAAAH!!! Quality journals won’t publish the boring shitte I work on, so it must be that everything they *do* publish stinks!!!!

    Like

  15. Pinko Punko Says:

    I think DM is rightly worried that the next crop of 28 year old journal editors will have received this sort of training and will propagate these types of feelings by considering scientists to only have value if their vitaes are larded with that stuff.

    Like

  16. BugSoc Says:

    Limited scope, missing controls, lack of any meaningful conclusions. Those are the papers we were trying to avoid for JC – and yes, people were picking some like that. You could argue that analyzing such a paper is an exercise in critical thinking but at the end I feel profoundly unsatisfied with the exercise. It’s more fun to analyze something flawed but ultimately of broad interest. Outside of JC of course there’s nothing wrong with narrowly focused papers, assuming the science is good.

    Like

  17. drugmonkey Says:

    You just combined limited scope with two *actual* problems. Nice sleight of hand but it doesn’t show why limited scope (of solid, properly interpreted science) is such a problem. Nor what the definition is.

    Like


  18. Yes, my experience with JC’s has been dept/program-run for-credit, and they have always had the C/N/S or top field specific journals requirement. This is the norm, based on my experience at several institutions.

    Becca is right that it’s useful to be able to pick apart a glamor mag publication in order to squash the idea that this is the only forum for excellent science (or that all articles in said journal consist of excellent science).

    But I think that can be done with a few papers from the glamor mags and not at the exclusion of other excellent science published elsewhere.

    Like

  19. Isis the Scientist Says:

    Dude, everyone knows that if you’re publishing in journals with IF<10, no one gives a fuck. You'll never get tenure, or grants, and you'll end up living in an ivy-covered cardboard box on some street corner.

    Like

  20. a Says:

    we had the same rule (no N/S/PNAS) for the much same reasons – nobody ever read the supp mat so the discussions always withered as we looked up to see if the point raised was covered in the supp mat.
    It also forced people to spend more time looking and thinking about whether something was of general interest rather than just picking the “hottest” thing

    Like

  21. UDbutnowUC Says:

    Isis, you’re on crack and this is totally not true.

    Like

  22. El Picador Says:

    Totes verdad. IF below 10 or 15 and you might as well not publish. Who reads that boring stuff anyway?

    Like

  23. scicurious Says:

    THIS. YES. THIS.

    The J Club I was referring to is dept. run, and not only are we limited to high IF journals, we’re limited to S/N/C, PNAS, Nature Neuroscience, etc. And it really tends to be much more about the ripping of the paper. It’s meant to be constructive, to teach us how to better construct our own papers to tell a complete story in an efficient way. It often comes across as ripping, though.

    Like

  24. Isis the Scientist Says:

    Exactly. I use things with IF<15 for bathroom reading. I browse the articles and then can use them to wipe my ass with.

    Anything other than that and you are wasting your time. I throw journals with IF<10 away with all those useless Sigma catalogs and societal newsletters.

    Like


Leave a comment