Repost: A Modest Proposal on Impact Factors

Talking about the h-index reminded me of how I really feel about citation metrics. This post went up on Sept 21, 2007.

---

People argue back and forth over whether Impact Factor of journals, the h-index, Total Cites, specific paper cites, etc should be used as the primary assessment of scientific quality. Many folks talk out of both sides of their mouths, bemoaning the irrelevance of journal Impact Factor while beavering away to get their papers into those journals and using the criterion to judge others. In this you will note people arguing the case that makes their CV look the best. I have a proposal:

I find the h-index to be very strange today

I like to use ISI’s web of knowledge thingy to keep track of who is citing our* papers. Often times I’ll pull up a few that I haven’t seen yet that are related to our work.

Fortunately, I don’t have consistent cause to review other performance metrics having to do with my pubs because the whole thing kind of gives me a headache.

But I do, now and again, look at the h-index and ponder it. I’m not quite grasping what it tells us, other than one’s longevity in science, but whatever. Seems to me that if you take a slice of your own approximate science-age cohort, then it might be at least somewhat meaningful.

I have a bunch of peers in my approximate subfields, of my approximate career status and, most importantly, who started publishing at approximately the same time as I did. This is the “hold all else equal” analysis, or at least as close as it comes.

I recently took a look at the citation reports of some folks that I think, in a very general sense, have been kicking my ass on the metrics. Particularly salient to me is the rate of publications flying out with their names on them, since I see them pass by TOC and PubMed topic alerts. And in many cases the graph of pubs per year on ISI’s web of knowledge confirms that impression. But the number of *citations* per year seems to feature a lot less variance than I would think.

Hmm I says to myself.

Then I look at the h-indices and find even less variance than I would have thought.

So now I’m back to trying to grasp what this measure really means. In an intuitive sense, I mean; I grasp the definition**.

If someone has a significantly larger number of papers, this should result in a higher h-index, right? I mean just going by the odds of what is going to lead to greater or fewer numbers of citations. If there is a longer length of time of publication, ditto, as they accumulate. And I grasp the notion that different subfields of science are going to be more or less active, citation wise. But when you start making comparisons between individual scientists who have approximately the same length of publishing history in approximately the same subfields, you should be able to use this h-index more accurately. It should say something meaningfully different. And I’m not seeing that right now.

Unless you argue that regardless of numbers of published articles that might be anywhere from 1.5-3 fold higher, the scientists in this grouping only manage to pump out about the same number of papers that have “top-X” amount of citation interest within the field?
__
*and I’m increasingly using this as a tool to track through literature that cites other specific papers in subfields. I’m getting more and more impressed with this as a time saver for my rapidly atrophying brain.

**The h-index as defined by the creator Hirsch: A scientist has index h if h of [his/her] Np papers have at least h citations each, and the other (Np − h) papers have at most h citations each.