Some citations are more equal than others…but this is invisible to the bean counters

September 20, 2012

As everyone enjoys themselves predicting their h-index using this new tool, it returns us to talking about the measurement of science and the bean-counting of citations.

For those who are new, citations of your academic papers are good, the more you have the better and all of this is well over 90% dependent on factors such as field size and vigor that have essentially nothing to do with the actual quality of your work itself.

Clear?

Nevertheless, within some approximation of a related set of peer investigators who publish in roughly the same journals you do….well, the number of citations you get may have something to do with how cool your work is. So there’s that.

My point for today is an excessively narrow one. There are plenty of reasons to show why citations cannot be compared to each other. But I mention to you a reason that will forever be transparent to any bean-counting attempt to quantify your paper quality.

A citation is not a citation.

Sometimes a paper is cited in a fluffy or peripheral way. Mentioned once in the Intro along with four other citations as a general point. Maybe even overgeneralizing and getting it a bit wrong.

Sometimes, a paper is cited in a fundamental, formative way. It is an essential background motivation or concept around which the present work is constructed.

The latter is fantastic and means the paper really had impact.

The former can be little better than a marker for being in the game and communicates very little other than the mere fact that you published a paper. That popped up on the first page of a PubMed search or something. Or happened to be lazily cascade-cited through a small thread of science.

The bean counting doesn’t give a rat’s patootie about which type of citation your paper received.

Advertisements

No Responses Yet to “Some citations are more equal than others…but this is invisible to the bean counters”

  1. Dr Becca Says:

    And sometimes a paper is cited for the purposes of saying “this is wrong.” Which, maybe “no such thing as bad publicity” applies here, but…I don’t know.

    Sorry so incoherent. Grant + wedding = brain mush.

    Like


  2. […] So what is a good h-index in the evolutionary biology field for roughly tenure-aged folks? How much variability is there in different fields? Does anyone give a shit? […]

    Like


  3. We are actually working on helping to fix this problem. And you can help too if you are interested!

    See

    From counting citations to measuring usage (help needed!)
    http://lemire.me/blog/archives/2012/03/20/from-counting-citations-to-measuring-usage-help-needed/

    We have an online form where you can help us by identifying which citations in your papers were most important:

    https://docs.google.com/spreadsheet/viewform?formkey=dHlDalFfR1AzTXpaRXA2WEVlRUF5b0E6MA#gid=0

    The goal is to build software that automatically detects “fundamental citations”.

    Some time this winter, we shall release some early results.

    Like

  4. Virgil Says:

    Also from the citing author’s point of view, don’t forget all those papers one has to cite because you fear your manuscript might get reviewed by someone who will get pissed if you didn’t cite them. Again not a “real” citation, more just paying lip service to someone in the field who would trash the paper if they felt ignored. My field has a lot of these “I’m important dammit!” types.

    Like


  5. […] I don't know what my h-index is, and I will not be taking the time to figure it out. It is stupid enough that this is used by some places as a metric for evaluation. But using an "algorithm" to predict the future is just stupid. […]

    Like

  6. AcademicLurker Says:

    I believe a link to this is in order.

    Like

  7. Cynric Says:

    Even more invisible: the times someone in the field describes your work in detail, but then cites someone more famous by accident.

    Not that I’m bitter.

    Like


  8. […] Scientists: overuse of antibiotics in animal agriculture endangers humans. Former FDA commissioner is among 150 scientists and 50 farmers calling on Congress to regulate antibiotic use Mysterious Underwater ‘Crop Circles’ Discovered Off the Coast of Japan GOP Candidate Wants To Eliminate Department Of Energy Because He Saw Officials ‘Sitting There Reading Books’ (I’m surprised the guy knows what a book is. Drum roll, please!) How can we solve the wild problem of science communication? H-index douchery Some citations are more equal than others…but this is invisible to the bean counters […]

    Like

  9. rs Says:

    agreed, but who has time to check what all those citations are for. I typically don’t bother to look. I move on and look at the next problem I have in my hand. For me once the article is published, it is gone from my hand and to the world and has its own life.

    Like

  10. Drugmonky Says:

    Virgil-

    That is fascinating. I’ve never once felt I had to include a citation in fear of reviewer trashing over a failure to be cited. I am curious as to what sort of evidence fuels your paranoia on this.

    Like


  11. […] Drugmonkey brings up an interesting point: Being cited can mean anything from “This is important” to “I wanted to make the point that this exists and it was the first paper that came up in Google Scholar.”  I can think of numerous reasons to pick a paper: […]

    Like

  12. abstrusi Says:

    Thanks for this post!

    What Virgil describes, happened to me once, not with a paper, but with my PhD-research proposal that was given to a researcher from my group (i didn’t know that he was the referee) who i forgot to cite. It would have been a ‘padding citation’, as there were so many more – for me more informative – articles to rather cite and it was not directly a proposal for the group and the topic.
    i really just forgot it, though, and i didn’t intend to piss him off, but he was incredibly much and he still is (this is almost one year ago). While i think it has of course a lot to do with specific personalities, i can only agree with Virgil on these type of ‘i’m important’-guys that seem to gather in specific fields where impact factors and such seem to be more important to people (maybe).

    Like


  13. […] Some citations are more equal than others. […]

    Like


  14. […] So what is a good h-index in the evolutionary biology field for roughly tenure-aged folks? How much variability is there in different fields? Does anyone give a shit? […]

    Like

  15. Eli Rabett Says:

    “I’ve never once felt I had to include a citation in fear of reviewer trashing over a failure to be cited. I am curious as to what sort of evidence fuels your paranoia on this.”\

    Indeed you have been a very fortunate little primate indeed. The key is when the review starts: The authors have not cited the seminal work of S, showing that they are absolutely clueless about the field.

    Like

  16. DrugMonkey Says:

    yeah….never really had that problem. fights over my *interpretation* of such “seminal work” perhaps but that’s about it.

    or maybe I am tone deaf to the signs of a reviewer mad about not being cited….

    Like

  17. whimple Says:

    maybe I am tone deaf to the signs of a reviewer mad about not being cited….

    The lack of enthusiasm to support your proposal of such an irritated reviewer when your grant eventually comes up for review doesn’t make a lot of noise.

    Like

  18. Laurent Says:

    Got once cited for an obvious point you could find no refs for (of course, because it is obvious). It was really a minor point “un”related to the main results. Wondered what would happen if I then begin to (self)cite this paper to justify the same point the same way. (hoping to hook people that cite without reading :). Now I have to wait 20 years to end the experiment… 🙂

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: