“It is time to help the pendulum of power swing back to favour the person who actually works at the bench and tries to discover things.”

August 15, 2007

Go read The mismeasurement of science Lawrence PA. 2007. Curr. Biol. 17: R583-R585.(ht: evolgen). Really. Right now. It is a fantastic commentary on the detrimental effect impact-factor chasing and the like has on the course of scientific investigation. The title of the post is the concluding sentence.

I get into discussions about this problem from time to time. Although I’ve perhaps touched on the issues in blog posts once or twice, I’ve never done the full critique. And now I don’t have to, thanks to Lawrence’s commentary. [Update 08/16/07: Go see David Colquhoun’s GoodScience site for more on this.]

Interesting observation on the commentary from The other 95%:

One problem I have thought about using standardized metrics to evaluate scientific progress is the aggregation of high citation potential research in only a handful o the “top” journals. … the trendy research of the day is published in high profile journal, while the more topical journals are left in the dirt. This prevents lower “impact” journals from escaping a certain impact factor range, making them less appealing to new researchers whose papers fit more in the journals interests

An interesting point about the inherent circularity of the system. One might expand on this to point out that to some extent the people defining “what makes a Nature or a Science publication” are the reviewers, pulled mostly from the pool of those currently publishing in these and similar journals. This means that there will be some additional circularity of the process. The problem arises when very important findings in a given subfield are categorically ignored on the basis of the subfield. Sure there is an editorial role to minimize this and the very top (aforementioned) journals are broadly based but it is still a problem. And it trickles down into “lower” journals. A specific example from my field would be the Journal of Neuroscience, the society journal of the gargantuan Society for Neuroscience which includes soup-to-nuts generality in its membership and annual meeting presentations. For the longest time they’d publish behavioral-lesion studies or intra-cerebral recording-behavior studies but not behavioral-pharmacology papers.

This is no doubt an over-generalization but feel free to go back over the last 15 years or so of the Journal and you will see that the standard for certain papers was relatively low and for other types of work relatively high. Do a lesion, characterize a behavior and bob’s yer uncle. Record single unit activity from a couple o’ dozen cells in a given region? Good ta go. Extensive dose-response functions for multiple antagonist/agonist challenges on a behavior? Few and far between. When you get down to specific comparisons of methodology and what might really be interpreted from studies, the differences are even more apparent. You can’t help but conclude that certain types of work were viewed as “J Neuroci” work and other techniques in neuroscience were judged categorically irrelevant for the journal. (Oi, that was a long polemic for a relatively minor point and in any case J Neurosci seems to be much better on this score recently. For one thing, a J Neurosci paper is looking a lot more like a Science paper these days-multiple techniques and too many experiments to actually be able to cover the methods adequately. )

Update: As luck would have it, a Nature News piece on the h-index.

5 Responses to ““It is time to help the pendulum of power swing back to favour the person who actually works at the bench and tries to discover things.””

  1. PhysioProf Says:

    In re Lawrence commentary: I do agree to some extent with what he said. Nonetheless, I think it is worth pointing out that it very easy for those who have amply reaped the spoils of a particular system to rail at the system once they have reached the top.

    Like

  2. drugmonkey Says:

    I would target my disdain in this mode at those who criticize the “system” while fully participating in it and perpetuating it. Editorials from Nature umbrella journals, for example. Pontificating from exEditors of these types of journals. Scientists who bemoan it and then go right on bragging on ImpactFactor and Nature/Science pub counts and using the same to dis others, for example.

    But the practical matter is that it is only those who are apparently successful in the “system” that can credibly criticize it. Otherwise, the response is inevitably “sour grapes”. Right? Don’t you subtly discount bitching and moaning from someone who can’t get a paper into Nature/Science/Cell or whatever? Someone who can’t get a NIH grant funded? The disgruntled postdoc who is having difficulty with the transition to independence ? At some level are you not thinking “Well, maybe they just aren’t good enough”.

    We all know this doesn’t mean the critiques aren’t valid. But it is a sort of inevitable psychological process…

    Like

  3. whimple Says:

    “Swinging the pendulum back” just isn’t going to happen. How many Deans of Medical Schools making the important personnel decisions can say the word “research” without also saying the word “dollars” in the same sentence? When you get a pile of 200 applicants for one faculty position, do you make the short list out of those with real creative potential, or is it those with the Cell, Nature and Science papers (CNS syndrome), and those well-connected to people with stacks of CNS papers?

    Lawrence gripes about how he was mistreated by Science — why did he send that paper to Science in the first place, instead of, say, Nucleic Acids Research? He (and everyone else) is counting on having his paper in Science give it instant credibility, instead of instant yukkiness from some “specialty journal” ***even though it would be the exact same paper***. In short, he doesn’t believe his work will be evaluated on the basis of the work itself, but rather on the basis of where it appears. Is this hypocrisy, or is it just participation in a self-perpetuating system?

    Like

  4. PhysioProf Says:

    “Is this hypocrisy, or is it just participation in a self-perpetuating system?”

    Yes.

    Like

  5. Piled Higher, Deeper Says:

    It isn’t hypocrisy necessarily. It is called being realistic. The impact (heh) of a CNS paper on your CV is unbelievably important, especially when you don’t have any. Who wouldn’t opt there if they thought there was a chance?

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: