ISI has two sets of citation books
December 20, 2007
Coturnix has the call:
there was a study the other day, in the Journal of Cell Biology, that seriously calls in question the methodology used by Thompson Scientific to calculate the sacred Impact Factor
A little bait from the article:
Articles are designated as primary, review, or “front matter” by hand by Thomson Scientific employees examining journals using various bibliographic criteria, such as keywords and number of references … Some publishers negotiate with Thomson Scientific to change these designations in their favor. The specifics of these negotiations are not available to the public, but one can’t help but wonder what has occurred when a journal experiences a sudden jump in impact factor.
Dude, you had me at “some publishers negotiate…”!
Because the impact factor calculation is a mean, it can be badly skewed by a “blockbuster” paper. … In a self-analysis of their 2005 impact factor, Nature noted that 89% of their citations came from only 25% of the papers published. When we asked Thomson Scientific if they would consider providing a median calculation in addition to the mean they already publish, they replied, “It’s an interesting suggestion…The median… would typically be much lower than the mean. There are other statistical measures to describe the nature of the citation frequency distribution skewness, but the median is probably not the right choice.”
And here we have a common refrain when talking about inconvenient facts about the IF. “Well, of course we know it is flawed but we’re going to keep right on using it as if it is not!” Nature, I’m looking at you, hypocrites.
Thomson Scientific explained that they have two separate databases—one for their “Research Group” and one used for the published impact factors (the JCR). We had been sold the database from the “Research Group”, which has fewer citations in it because the data have been vetted for erroneous records. “The JCR staff matches citations to journal titles, whereas the Research Services Group matches citations to individual articles”, explained a Thomson Scientific representative. “Because some cited references are in error in terms of volume or page number, name of first author, and other data, these are missed by the Research Services Group.” When we requested the database used to calculate the published impact factors (i.e., including the erroneous records), Thomson Scientific sent us a second database. But these data still did not match the published impact factor data. This database appeared to have been assembled in an ad hoc manner to create a facsimile of the published data that might appease us. It did not.
It does not appease me and it should not appease you either, DearReader.
… and take Coturnix’s closing advice:
And several other bloggers seem to agree, including Bjoern Brembs, The Krafty Librarian, Eric Schnell, Peter Suber and Stevan Harnad who each dissect the paper in more detail than I do, so go and read their reactions.