A Modest Proposal on Impact Factors

September 21, 2007

People argue back and forth over whether Impact Factor of journals, the h-index, Total Cites, specific paper cites, etc should be used as the primary assessment of scientific quality. Many folks talk out of both sides of their mouths, bemoaning the irrelevance of journal Impact Factor while beavering away to get their papers into those journals and using the criterion to judge others. In this you will note people arguing the case that makes their CV look the best. I have a proposal:

Instead of focusing on the journal’s Impact Factor what we really need to focus on is how a given paper performs relative to expectation. After all, you should publish your paper in the most appropriate forum, right? Fit on length, completeness, desired audience, methodologies employed, scientific subfield, etc. So if your paper is in the right journal, what we really need to know is how it performs, citation-wise, versus the average for the journal. If you are in a very large, active and citation-generating field you should be judged against this metric, rather than against smaller and slower fields.

So we need a d-index (If Hirsch can call his the h-index, I’m happy to follow suit). This would be the number of actual citations for your paper expressed as a z-score for the distribution of citations in the journal over the previous, oh, ten-year interval.

Issues:

“but, but….I got my paper into Science. That proves it is good!” Nope, sorry, “true peer review starts after publication”.

“but it will change over time as cites start to accumulate”. So? A paper’s true worth should only be determined over time as parts of it are replicated and/or applied to other scientific questions. This is what any cite metric is about anyway. The current focus on journal Impact is simply a very indirect predictor.

“this will encourage journal-slumming” No it won’t. All other influences which determine where papers are submitted are not going to magically disappear.

UPDATE 09/25/07: From the article whimple identified in the comments a description of cite distribution “from the theory section of the SPIRES database in high-energy physics”

About 50% of all papers have two or fewer citations; the average number of citations is 12.6. The top 4.3% of papers produces 50% of all citations whereas the bottom 50% of papers yields just 2.1% of all citations.

This also put me back on the track of two relevant papers on citation distributions from Per O Seglen (1992; 1994)

About these ads

6 Responses to “A Modest Proposal on Impact Factors”

  1. bikemonkey Says:

    I think you need to recognize the incredible skew in cites, not just for the very highest of impact journals but for the run of the mill too. How about using position relative to median and/or quartile rank?

    anyone know what the mode is for C, N or S paper cites by the bye?

    Like

  2. whimple Says:

    This is a fun article on the topic.
    Nature 444, 1003-1004 (21 December 2006) | doi:10.1038/4441003a; Published online 21 December 2006
    I particularly like where they establish that the metric of “number of papers per year” has the same predictive power of scientific quality as does the location in the alphabet of the author’s name.

    Like

  3. drugmonkey Says:

    Finally ran across some Nature editorial comments I was remembering when I wrote this entry.

    The most cited Nature paper from 2002−03 was the mouse genome, published in December 2002. That paper represents the culmination of a great enterprise, but is inevitably an important point of reference rather than an expression of unusually deep mechanistic insight. So far it has received more than 1,000 citations. Within the measurement year of 2004 alone, it received 522 citations. Our next most cited paper from 2002−03 (concerning the functional organization of the yeast proteome) received 351 citations that year. Only 50 out of the roughly 1,800 citable items published in those two years received more than 100 citations in 2004. The great majority of our papers received fewer than 20 citations.

    Like


  4. [...] 11/20/07: Of course I’d forgotten I already created the d-index.] Posted by drugmonkey Filed in Day in the life of Drugmonkey, [...]

    Like


  5. [...] about the h-index reminded my of how I really feel about citation metrics. This post went up on Sept 21, 2007. People argue back and forth over whether Impact Factor of journals, the h-index, Total Cites, [...]

    Like

  6. Jason Riedy Says:

    There are entirely different methods possible. Treat citations as a social network, and suddenly you have zillions of only partially understood possibilities.

    Betcha a publication comparing them would rank highly on many. ;)

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 42 other followers

%d bloggers like this: