On defining productivity of the NIH grant award
September 6, 2012
This comment at Rock Talk…
NIH should stop allowing multi-awarded PIs to double count publications on multiple grants. This leads to inflation in productivity, and is one reason well funded can fool study sections into thinking they are more productive than they really are. If a publication is counted on multiple grants, its impact on each should be fractionally decreased so that its total impact is the same as the one publication on a single grant PI. Unless council does this, they will not have an accurate indication of whether the increased funding is actually leading to commensurate increase in productivity.
…has the right of it.
It took me about five minutes into reading my very first competing-continuation application when I first sat on a study section to realize that many people are very generous about crediting the grants they happen to be holding on the papers that they submit.
It took me perhaps half a day of study section discussion to figure out why they do this. Because “this amazingly productive researcher” has serious value in the discussion of a grant proposal. And this viewpoint on the part of a reviewer need not be objective in any way. All that is required is the thought that “gee, I see a lot of papers coming out of Dr. Squirrel’s laboratory”. And if ol’ Dr. Squirrel happens to fill up the first page of PubMed hits with papers from the current year and several pages of publications within the past 5 years, then everyone sitting around the table who cares to check will start nodding in agreement.
This is particularly important when it is a competing continuation applications. This type of application (asking for another 5 year interval of support for a project that is already underway) has an explicit section for detailing productivity. Nowadays, I think most people are on board with the idea that they need to list specific NIH grant numbers on each paper (sometime ago it was reasonably common to just say “NIH support” or “NIMH support” or something). So the progress report list better be of papers that mention the grant under discussion. So the smart PI is thinking all along about this list and how long she would like it to be. So she cites as many of her grants as possible on each publication.
And nobody checks.
Well, this isn’t strictly accurate. I have heard people try to reign in a comment about “wonderful productivity under this award” with a close analysis of the listed publications. To point out where a publication appearing in print three months after the start of the award couldn’t possibly have been conducted with the support of that particular award. To show where the attribution of a paper to this particular grant, given the other attributed grants, was an overreach of epic scale. To argue that even if two or three grants might have contributed equally to the paper, it was necessary and fair to divide by two.
I don’t think I’ve ever seen this actually work. I don’t think I’ve once seen a reviewer who stated “wonderfully productive” fully grasp what a critic was saying and reverse his/her opinion.
What I wonder about is the degree to which overall culture on study section can change with respect to this. (And, per usual, I throw this out to my readers for their respective experiences.)
My thought is that this sort of take on “productivity” is entirely dictated by the grant and seniority status of the reviewer. One-grant noobs are absolutely ENRAGED by this seeming disparity. Established PIs who do exactly this same thing in their own grant management strategy act like they don’t know what the youngsters are talking about.
The question is how the various Councils and POs will view this whenever “productivity” becomes an issue.
And I have to tell you, Dear Reader, my confidence that various Program types understand what is going on with this sort of gaming is not very high.
__
Additional Reading:
Your Grant in Review: Productivity on Prior Awards
Musing on NIGMS’ grant performance data
Another Look at Measuring the Scientific Output and Impact of NIGMS Grants
Productivity Metrics and Peer Review Scores
Mapping Publications to Grants
Comparing performance of within-payline and “select pay” pickup NIH grants at NIAID
DrugMonkey 
September 6, 2012 at 7:53 am
The section I was on definitely looks at this, but they don’t do it across the board. I would say maybe half the reviewers didn’t check and the other half specifically did. Of course if NIH ASKED for that information on the Biosketch (list funding sources that are attributed for each paper), that would make it both easy and suggest that it should be stressed.
Usually for bigwigs, what happens is that “well, these papers were double counted, but they are sooooo significant THAT is what really counts”. I do know of a situation [hearsay] where a 3 R01 person got busted down to two, because every paper in the lab was triple dipping. This was a lab that only published C/N/S papers, but the section I think had enough that said that the productivity wasn’t enough for one of the grants (because all three grants were attributed on say the 6 C/N/S papers).
LikeLike
September 6, 2012 at 7:56 am
Adding, I have observed that some PIs will address this issue right up front because they are double dipping all papers, but they do so when the math is in their favor (dividing by 2 indicates there still isn’t an issue). Of course dividing by grant isn’t quite fair when say biggie has 2 600K/year R01s and this is supposedly compared to not even a full modular.
LikeLike
September 6, 2012 at 7:57 am
wasn’t there a researcher a few years ago who got hit for fraud after citing grant support from both NIH and DoE (or some other pair of federal funding bodies) on a publication?
LikeLike
September 6, 2012 at 8:24 am
When universities are requiring us to support 90% of our salary on grants, every grant is contributing to that paper in some way as every grant is contributing to your salary. Unless you are truly partitioning your time in such a way as NIH mandates (eg. for those 3 calendar months you only work on that project and only get paid by that grant), I see no way to not cite every grant that supports your salary and still be truthful.
LikeLike
September 6, 2012 at 8:39 am
Time can be apportioned down to the day- “months” is just a shorthand for fraction of the total.
For grants that are supposed to be entirely non-overlapping, there can be some issues when a paper is claimed to be supported by both, but I have never heard of a fraud case. Places like NSF would be pissed if they think you have overlapping funding. They really don’t like that.
LikeLike
September 6, 2012 at 8:40 am
This is fucking bananas. Who sits on study sections? GOP economists?
Maybe I’m wrong, but it seems like it is the norm that you cannot discretely assign separate publications from the same lab to specific grants. But if the output you are flogging for each of n grants is X, then how can a study section ethically consider your output for a particular grant to be anything other than X/n?
OTOH, if nothing else, this blog has convinced me that study section is like any other peer review process, in which any specific criteria or scoring is a meaningless and post hoc method to justify a subjective judgement: I like / I don’t like.
LikeLike
September 6, 2012 at 8:54 am
Exactly, miko. WHAT IS THE PROBLEM with the equation, productivity = pubs / grant dollars? Do we need Bill Clinton to get in there to teach study sections some arithmetic?
It is completely and absolutely absurd to allocate papers to specific grants. My grants are all related in some way: for instance, a technique whose development was paid for in grant A is used for experiments X, Y and Z, some of which are described in grant A, some of which are described in grant B. The papers describing the results of these experiments all cite both grants A and B. It’s somehow wrong or fraudulent to submit a grant renewal saying “I got three papers out of grant A”?
The idea that grant reviewers should solve the question of determining productivity by allocating attention to the idiotic details about what exact grant dollar funded what exact experiment is a typical bureaucrat’s solution. It entirely misses the forest for the trees – especially considering that the simple equation given above solves the problem almost effortlessly.
LikeLike
September 6, 2012 at 9:10 am
I ask this of all of you here today: please kill me if my mind ever becomes warped enough to think anything about the NIH makes sense, from their horrific 90s web site with its circular links, their illiterate forms and general disregard for the English language, their sociopathic “dynamic” PDFs, the stupid amount of money and human effort wasted on incoherent and needlessly Byzantine decision making processes… it’s like the whole fucking thing is a Terry Gilliam fever dream.
LikeLike
September 6, 2012 at 9:22 am
There is no issue about double dipping grants, but it goes to your total productivity. However, if a paper describing your technique is supported by one grant, that can be attributed to that grant. If you use the technique in another paper you can put this paper on another grant. Of course there is flexibility. The issue is if you have 12 papers and 3 R01s, your productivity per R01 is not 12 papers, it is 4. This is the issue. This is why it should be checked. You may not buy miniprep kits on every single grant, but you are also supposed to somewhat segregate money- i.e. if people are supported by multiple grants, then part of that person is on each of the grant. It is not too crazy to ask for a little bit of accounting. Drug Monkey is making the point that double and triple dipping should probably be checked more than it is. I know it is looked at in some sections for sure. And I know that sections I have been on have not been 100% post hoc arguments, I would say the section was pretty fair, but it wasn’t perfect.
LikeLike
September 6, 2012 at 9:24 am
there it is:
http://www.nature.com/news/duplicate-grant-case-puts-funders-under-pressure-1.9984
“Yet in a 2010 paper (S. C. Roy et al. ACS Nano 4,1259–1278; 2010) he openly acknowledged both the NSF and ARPA-E for supporting the same work. That year, the DOE inspector- general spotted the similarity between the grants, the NSF began its investigation, and Grimes resigned his university position.”
LikeLike
September 6, 2012 at 9:26 am
When universities are requiring us to support 90% of our salary on grants, every grant is contributing to that paper in some way as every grant is contributing to your salary. Unless you are truly partitioning your time in such a way as NIH mandates (eg. for those 3 calendar months you only work on that project and only get paid by that grant), I see no way to not cite every grant that supports your salary and still be truthful.
Agreed.
LikeLike
September 6, 2012 at 9:31 am
from their horrific 90s web site with its circular links
The NIH website is truly a marvel of human ingenuity. I can still remember starting out as a new PI how long I spent clicking around trying to find the forms for the basic RO1 application and going in circles.
Good times.
LikeLike
September 6, 2012 at 9:34 am
Seriously. There are fucking cupcake stores in this town that have put more effort into their web site functionality.
LikeLike
September 6, 2012 at 9:40 am
Until and unless funding agencies forbid listing multiple grants for the same paper, it is only rational (and not unethical as long as all the grant proposals bore some relation to the paper) to do so. It sounds reasonable to divide by N in assessing productivity if one has to quantify productivity… but none of the agencies I deal with (I am a non-biomedical type) requires such a quantification so it never really comes up to this degree. Peer review is not that rational a process – it really can’t be I don’t think – and so trying to make it really quantitative and precise is always going to be a losing battle.
LikeLike
September 6, 2012 at 9:42 am
Furthermore the same double counting that helps PIs (when reviewers don’t divide by N) is also in the interest of funding agencies and PMs, in that it makes their programs look more productive to their bosses and ultimately Congress. So I don’t imagine it will ever be outlawed.
LikeLike
September 6, 2012 at 10:49 am
I dunno. I have a paper in submission that was supported in part by several grants. Are we not supposed to credit them all?
LikeLike
September 6, 2012 at 11:28 am
Double *dipping* implies something a little different to me- that of doing one grants worth of work and getting two grants to do it. I tend to credit that people are not quite this craven. Rather, they have an output for each project but that they are trying to make each one look better than it is.
One corrosive and lasting effect is that it sets up the expectation of what someone should be able to accomplish with one grant. So even for new proposals, there is a possibly unfair expectation of scope.
LikeLike
September 6, 2012 at 12:29 pm
It’s even worse when a C/N/S paper has multiple PIs each with multiple grants. There you can find six, and upwards, grants acknowledged even when the contribution of a particular PI is “made editorial suggestions”. Perhaps the noobs should resort to “collaborating” among their own kind to jack up their perceived productivity. Hell, someone could probably get rich with a business matching up PIs to help maximize their sleight of hand productivity.
LikeLike
September 6, 2012 at 12:49 pm
“craven”, huh? has DM been reading GRRM?
LikeLike
September 6, 2012 at 12:51 pm
why all these discussons about IF, publication/dollar reminds me one of the most interesting statement of qunatum thoery that by the very act of watching, the observer affects the observed reality.
LikeLike
September 6, 2012 at 2:21 pm
I’m with miko, this shit is bananas…
B
–
A
–
N
–
A
–
N
–
A
–
S
If the grant paid a tiny % of my salary while I was writing the paper, or it paid for the toilet paper for the tech who did the cell culture on the paper, then the grant gets thanked on the paper! Does this mean some papers end up thanking 10 grants? Yes. Does it mean some grants get thanked on papers to which they made a minimal contribution? Yes. Does it mean when you co-author with someone from out of town, and they ask to include their grant on the paper, you tell them no? That would be douchey.
Unless they figure out a way of salami slicing this shit up (i.e. hiring a shitte-lode more administrators to oversee this stuff and assign percentages of efforts to single figures of papers), che sera sera!
LikeLike
September 6, 2012 at 2:42 pm
“It is not too crazy to ask for a little bit of accounting.”
The way to do it is to have the NIH grant forms ask for total research funding from all sources in the last, say, 5 years. Then ask reviewers to evaluate productivity (evident from the papers listed in the biosketch) in relation to past funding. If it’s made explicit this way, that would somewhat increase the consistency of the use of this factor in grant reviews.
Of course, everyone evaluates productivity differently, so the whole damn thing is subjective anyway, so why bother with even this bit of “accounting”? It would just waste everyone’s time, like everything else the NIH asks for on grant applications (“Is your building a historical building?” WTF?).
LikeLike
September 6, 2012 at 3:39 pm
i.e. hiring a shitte-lode more administrators to oversee this stuff….
Please no.
LikeLike
September 6, 2012 at 6:49 pm
Just to leave an anecdote, some PIs seem to list just about everything in the progress reports (or competing renewals). 10-20+ publication might be listed, but all, of a sudden, they have none that are grant supported when asked to provide PMCIDs to show compliance with public access.
LikeLike
September 7, 2012 at 3:28 am
Agreed it’s a problem. disagree that imposing some half-arsed metric is going to be the solution to that problem. Reviewers should be encouraged to consider grant dollars vs papers published when forming their opinions, but this must still rely on a mostly subjective appraisal of study size, quality, relevance to grant etc. It cannot be distilled down to some lazy and dangerously arbitrary ratio of pubs to grant numbers/dollars.
LikeLike
September 7, 2012 at 6:58 am
Without “lazy and dangerously arbitrary” criteria, science funding/publishing/hiring wouldn’t have no criteria at all, so let’s not limit our imaginations. OTOH, no specific metric is necessary to require PIs to prominently disclose the entire research budget they had available when touting their productivity.
LikeLike
September 7, 2012 at 9:19 am
KILL THE RICH!!!!11!!11!! EAT THEIR CORPSES!!!!11!1!1!!!
LikeLike
September 8, 2012 at 4:43 am
The CPP comment bot experiment is failing… deployment of stock statements getting less rather than more relevant over repeated trials.
LikeLike
September 9, 2012 at 12:04 pm
For those of us engaged in multi-disciplinary efforts, the assignment of any particular publication to a single grant is not always possible. Even the development of a new technique may find its roots in more than a single line of experimentation (read: RO1). This argument applies a fortiori to starting a new line of experimentation: if not covered by the aims of a specific RO1, where to assign the credit when things begin to work out? It might make sense to assess productivity against total $$ amount available, and not obsess about which grant to credit for a particular paper.
LikeLike