Some citations are more equal than others…but this is invisible to the bean counters
September 20, 2012
As everyone enjoys themselves predicting their h-index using this new tool, it returns us to talking about the measurement of science and the bean-counting of citations.
For those who are new, citations of your academic papers are good, the more you have the better and all of this is well over 90% dependent on factors such as field size and vigor that have essentially nothing to do with the actual quality of your work itself.
Clear?
Nevertheless, within some approximation of a related set of peer investigators who publish in roughly the same journals you do….well, the number of citations you get may have something to do with how cool your work is. So there’s that.
My point for today is an excessively narrow one. There are plenty of reasons to show why citations cannot be compared to each other. But I mention to you a reason that will forever be transparent to any bean-counting attempt to quantify your paper quality.
A citation is not a citation.
Sometimes a paper is cited in a fluffy or peripheral way. Mentioned once in the Intro along with four other citations as a general point. Maybe even overgeneralizing and getting it a bit wrong.
Sometimes, a paper is cited in a fundamental, formative way. It is an essential background motivation or concept around which the present work is constructed.
The latter is fantastic and means the paper really had impact.
The former can be little better than a marker for being in the game and communicates very little other than the mere fact that you published a paper. That popped up on the first page of a PubMed search or something. Or happened to be lazily cascade-cited through a small thread of science.
The bean counting doesn’t give a rat’s patootie about which type of citation your paper received.
DrugMonkey 
September 20, 2012 at 9:28 am
And sometimes a paper is cited for the purposes of saying “this is wrong.” Which, maybe “no such thing as bad publicity” applies here, but…I don’t know.
Sorry so incoherent. Grant + wedding = brain mush.
LikeLike
September 20, 2012 at 10:14 am
[…] So what is a good h-index in the evolutionary biology field for roughly tenure-aged folks? How much variability is there in different fields? Does anyone give a shit? […]
LikeLike
September 20, 2012 at 12:08 pm
We are actually working on helping to fix this problem. And you can help too if you are interested!
See
From counting citations to measuring usage (help needed!)
http://lemire.me/blog/archives/2012/03/20/from-counting-citations-to-measuring-usage-help-needed/
We have an online form where you can help us by identifying which citations in your papers were most important:
https://docs.google.com/spreadsheet/viewform?formkey=dHlDalFfR1AzTXpaRXA2WEVlRUF5b0E6MA#gid=0
The goal is to build software that automatically detects “fundamental citations”.
Some time this winter, we shall release some early results.
LikeLike
September 20, 2012 at 6:23 pm
Also from the citing author’s point of view, don’t forget all those papers one has to cite because you fear your manuscript might get reviewed by someone who will get pissed if you didn’t cite them. Again not a “real” citation, more just paying lip service to someone in the field who would trash the paper if they felt ignored. My field has a lot of these “I’m important dammit!” types.
LikeLike
September 20, 2012 at 9:23 pm
[…] I don't know what my h-index is, and I will not be taking the time to figure it out. It is stupid enough that this is used by some places as a metric for evaluation. But using an "algorithm" to predict the future is just stupid. […]
LikeLike
September 21, 2012 at 4:39 am
I believe a link to this is in order.
LikeLike
September 21, 2012 at 5:33 am
Even more invisible: the times someone in the field describes your work in detail, but then cites someone more famous by accident.
Not that I’m bitter.
LikeLike
September 21, 2012 at 11:45 am
[…] Scientists: overuse of antibiotics in animal agriculture endangers humans. Former FDA commissioner is among 150 scientists and 50 farmers calling on Congress to regulate antibiotic use Mysterious Underwater ‘Crop Circles’ Discovered Off the Coast of Japan GOP Candidate Wants To Eliminate Department Of Energy Because He Saw Officials ‘Sitting There Reading Books’ (I’m surprised the guy knows what a book is. Drum roll, please!) How can we solve the wild problem of science communication? H-index douchery Some citations are more equal than others…but this is invisible to the bean counters […]
LikeLike
September 21, 2012 at 4:52 pm
agreed, but who has time to check what all those citations are for. I typically don’t bother to look. I move on and look at the next problem I have in my hand. For me once the article is published, it is gone from my hand and to the world and has its own life.
LikeLike
September 22, 2012 at 1:30 am
Virgil-
That is fascinating. I’ve never once felt I had to include a citation in fear of reviewer trashing over a failure to be cited. I am curious as to what sort of evidence fuels your paranoia on this.
LikeLike
September 22, 2012 at 1:29 pm
[…] Drugmonkey brings up an interesting point: Being cited can mean anything from “This is important” to “I wanted to make the point that this exists and it was the first paper that came up in Google Scholar.” I can think of numerous reasons to pick a paper: […]
LikeLike
September 23, 2012 at 11:08 pm
Thanks for this post!
What Virgil describes, happened to me once, not with a paper, but with my PhD-research proposal that was given to a researcher from my group (i didn’t know that he was the referee) who i forgot to cite. It would have been a ‘padding citation’, as there were so many more – for me more informative – articles to rather cite and it was not directly a proposal for the group and the topic.
i really just forgot it, though, and i didn’t intend to piss him off, but he was incredibly much and he still is (this is almost one year ago). While i think it has of course a lot to do with specific personalities, i can only agree with Virgil on these type of ‘i’m important’-guys that seem to gather in specific fields where impact factors and such seem to be more important to people (maybe).
LikeLike
September 24, 2012 at 6:51 pm
[…] Some citations are more equal than others. […]
LikeLike
September 25, 2012 at 3:41 am
[…] So what is a good h-index in the evolutionary biology field for roughly tenure-aged folks? How much variability is there in different fields? Does anyone give a shit? […]
LikeLike
September 25, 2012 at 7:32 am
“I’ve never once felt I had to include a citation in fear of reviewer trashing over a failure to be cited. I am curious as to what sort of evidence fuels your paranoia on this.”\
Indeed you have been a very fortunate little primate indeed. The key is when the review starts: The authors have not cited the seminal work of S, showing that they are absolutely clueless about the field.
LikeLike
September 25, 2012 at 7:53 am
yeah….never really had that problem. fights over my *interpretation* of such “seminal work” perhaps but that’s about it.
or maybe I am tone deaf to the signs of a reviewer mad about not being cited….
LikeLike
September 26, 2012 at 3:19 am
maybe I am tone deaf to the signs of a reviewer mad about not being cited….
The lack of enthusiasm to support your proposal of such an irritated reviewer when your grant eventually comes up for review doesn’t make a lot of noise.
LikeLike
September 28, 2012 at 11:19 am
Got once cited for an obvious point you could find no refs for (of course, because it is obvious). It was really a minor point “un”related to the main results. Wondered what would happen if I then begin to (self)cite this paper to justify the same point the same way. (hoping to hook people that cite without reading :). Now I have to wait 20 years to end the experiment… 🙂
LikeLike