I’ve had a few interactions lately that have led to some pondering on the attribution of academic credit for papers. It all starts with the hilarious gyrations that promotions and tenure committees occasionally, maddeningly, go through.
As most of my readers are aware, in biomedical sciences we have a system in which there are four key elements to placing credit for a given academic paper.
1.) The first author. This person is generally a trainee and generally assumed to be the person who did “most” of the real work on the paper. This can be defined in many ways, from conducting the bulk of the experiments to the most important experiments to the drafting of the manuscript. Defined, that is, by the research team itself when determining who is deserving of the first authorship. Once the paper has been published, the assumption about the role of the first author is more nebulous but no less firm in attributing the academic credit.
2.) The senior author. Often the last author, often the primary mentor of the first author and often the PI of the grants identified as supporting the project.
3.) The “communicating author”. Most often the senior author, less frequently the first author and very infrequently someone else*. If the senior author, this is just a reinforcing stamp on his/her seniority, particularly if there are several relatively senior people with their own laboratories contributing. If the communicating author is the first author, this can be an indication from the research team that the project is really all under the intellectual domain of the postdoc or graduate student in question. The senior author is saying “no, really, it was all my brilliant postdoc and she should get all the credit. ps, email her for reagents or mouse lines, not me”.
4.) The grants identified as supporting the project and, by extension, the PIs of those grants.
P&T committees frequently find themselves parsing academic credit schemes not just from the biomedical perspective, but also from alternate academic traditions in which the number of other authors matters more than it does in biomedical disciplines, where the senior author is the first author, where single-authorship is important….or there is variance in other minutia. This can in itself be infuriating, after all, how hard is it to recognize that the tradition you trained in is only one of many equally arbitrary crediting schemes?

Read the rest of this entry »

I’ve had a few interactions lately that have led to some pondering on the attribution of academic credit for papers. It all starts with the hilarious gyrations that promotions and tenure committees occasionally, maddeningly, go through.

As most of my readers are aware, in biomedical sciences we have a system in which there are four key elements to placing credit for a given academic paper.

1.) The first author. This person is generally a trainee and generally assumed to be the person who did “most” of the real work on the paper. This can be defined in many ways, from conducting the bulk of the experiments to the most important experiments to the drafting of the manuscript. Defined, that is, by the research team itself when determining who is deserving of the first authorship. Once the paper has been published, the assumption about the role of the first author is more nebulous but no less firm in attributing the academic credit.

2.) The senior author. Often the last author, often the primary mentor of the first author and often the PI of the grants identified as supporting the project.

3.) The “communicating author”. Most often the senior author, less frequently the first author and very infrequently someone else*. If the senior author, this is just a reinforcing stamp on his/her seniority, particularly if there are several relatively senior people with their own laboratories contributing. If the communicating author is the first author, this can be an indication from the research team that the project is really all under the intellectual domain of the postdoc or graduate student in question. The senior author is saying “no, really, it was all my brilliant postdoc and she should get all the credit. ps, email her for reagents or mouse lines, not me”.

4.) The grants identified as supporting the project and, by extension, the PIs of those grants.

P&T committees frequently find themselves parsing academic credit schemes not just from the biomedical perspective, but also from alternate academic traditions in which the number of other authors matters more than it does in biomedical disciplines, where the senior author is the first author, where single-authorship is important….or there is variance in other minutia. This can in itself be infuriating, after all, how hard is it to recognize that the tradition you trained in is only one of many equally arbitrary crediting schemes?

What kills me though is when they look at the senior BigCheez biomedical types who are up for promotion to Full or “University Professor of Swangage” or some such and say “Dude, there is no way you actually wrote the fifteen manuscripts you listed in the past year, we gotta divide this up somehow”, while simultaneously looking at some poor schmuck Assistant Professor or Research Professor of -ology or whatnot and saying “Hrm, hrm, hrm. You have this senior author on all of your papers so we simply cannot, CANNOT I tell you, infer that you are operating as an independent investigator and you must simply be the tool of the BigCheez”. A lovely little Catch-22.

Next, I’ve been pondering the attribution of credit from the grant-game side of the fence. Remember the post at the NIGMS blog which presented data on the number of publications per investigator, broken down by the total amount of NIH grant funding? Look at the second figure and the jump between one grant (~$200K in direct costs) and two grants (~$400K). The mean number of publications goes up by maybe two but look at the intraquartile range. The 75th percentile nearly doubles. I notice that the people claiming that productivity doesn’t scale with the number of awards are talking mostly about the mean, not the range. And the trendline is ascending, not leveling off (as does the mean). Part of this may be, however, because some PIs, particularly those with reasonably basic research approaches, just go ahead and list every grant they have in hand on each and every paper that goes out.

The reason to do this, of course, is that when the NIH award is up for competitive renewal the PI lists all the resulting publications with the goal of looking “incredibly productive”. And this works out exactly as planned, in my grant reviewing experience. The reviewer already has a Gestalt impression, in many cases, that the PI in question is “productive” because they notice the papers keep coming out. They review the Biosketch and the Progress Report listing the publications associated with the grant in question and say to themselves “wow! looks great”. A perception of fabulous productivity covers a lot of ills in the actual grant proposal itself. Let me tell you though, the vast majority of reviewers are not calculating machines who attribute fractional credit based on how many grant awards are listed in the Acknowledgements section. Nor do they** look at the content of each publication to determine exactly which figures were attributable to the grant under current review. And of course, this reality screws the PI who has only the single award for whom each and every manuscript really was supported by the grant in question in its entirety.

I arrive at the ethical question from the PI perspective. What is fair? What’s the minimum amount of time an award is active before it can be put on a paper? Or what sort of expenditure? Do you have to have paid some sort of research bill from it or is staff time enough? What about if it is only while you are revising a previously-submitted MS?

We all have to make our own calls, of course, and the point I made above about making the competing continuation look as good as possible is no joke. My standard is a loose one: I should have some plausible connection between the goals of the grant in question and the data in the manuscript. So yes, I have definitely submitted manuscripts that did not list all of my active awards. And I think that I’ve listed awards on papers only for which the connection was reasonable…but I’m sure opinions would vary on that one.

But how about start dates? This intrigues me. I have definitely seen competing continuation applications come through listing support for an article that as published 6-9 mo after the start of the award. In society level journals that take nearly this long from original submission to publication of the article. So I know dollars to doughnuts that no real research expenditures were being made on the immediately past interval of support. But the team may have been polishing up or revising the manuscript, right?

Personally, I don’t think I have any minimum standard for the dollar contribution of a given award to the manuscript in question. My effort is primarily brain effort these days so, for example, if I’m working on writing or revising or fighting reviewers/editors, then I’m working on that manuscript. So technically, if the award is active and I’m having my effort paid from it, then that support is contributing. If I come up with some brilliant ideas that generalize across several of my research programs, that is a contribution. If a member of my lab is working up an assay or technique that we can apply to multiple projects, that contributes. I see no practical way to assess how much of a contribution has been made. So it seems like the brightest line we can draw is to determine if the grant is active before or after the manuscript has been accepted for publication.

The single-award laboratory is still considerably disadvantaged by this because of the perception factor. Really, if the grant score hinges on “productivity”, we really should be able to divide the pubs by the number of awards or the amount of $$ or something. But I’m telling you, the reviewer mind can’t really do this.

The perception of “amazing productivity” of the PI / lab is just too strong.

_
*a secondary “senior author”, most typically. Another PI who appears as an author, perhaps in next-to-last position. Might be PI of one of the awards listed as supporting the project or supervisor of the second author.

**except the occasional jerk trying to make a point whom the rest of the study section promptly ignores as a nutter…

sigh.

Our very dear blog friend writedit is mads with Your Humble Narrator.

Perhaps DM can restrict these sorts of discussions to his various for-profit plots in the blogosphere, where he is welcome to take on all comers.

So, alas, I simply must, MUST I tell you, take the discussion over here…

If you haven’t been following along, start with this comment from the walrus is paul, a self-identified NIH SRO that has been frequenting writedit’s blog of late. The triggering comment was walrus referring to a query about idea thievery in grant review as “tin-foil hat paranoia”. I disagreed that this qualifies as the “tin-foil hat” variety of paranoia. My argument is that usually this term is reserved for the most extreme beliefs in conspiracies and weird events for which there is no plausible evidence or rationale in the land of the sane. I also pointed out that this epithet is usually not dictated by the number of people who maintain a particular belief but rather the nature of that belief.

In this particular case, I argue that we hear assertions of idea stealing now and again from angry applicants. I hear this in person and I’m pretty sure the triggering comment (UPDATE: and additional comment making it clearer that this is indeed what the person meant) is not the first time this has surfaced in the blogosphere. Note that my recognition of this tells you absolutely nothing about whether I personally credit each such accusation. OTOH, being the good student of human behavior that I am, I also assert that yes, it is highly likely that there are or have been some incidents where the content of a grant application influenced the scientific conduct of the reviewer.

The proper answer is that of course somewhere, sometime an idea has been lifted from a grant application to the advantage of the reviewer and disadvantage of the applicant. It seems unlikely that it happens anywhere near as often as paranoid applicants who comment online would like us to believe, however. This latter derives, IMNSHO, from a laughable conceit on the part of many scientists that they are uniquely brilliant snoflakes and nobody else could possibly have the same exact ideas from reading the same literature and being in the same subfield.

The more interesting question is the degree to which subconscious influence operates despite the reviewer’s best intentions not to benefit from reviewing someone else’s grant proposal. IMO, of course

To ignore this reality and pretend that all possible participants in grant review are as pure as the driven snow, and that the structural features of the grant review meeting itself


we are REQUIRED to make a speech at the start of every review meeting about ethical conduct which includes a section on ” the ideas presented in applications are not to be pilfered”.

the walrus (and writedit) thereupon asserted that in their professional (and online?) capacities they have never, ever run across any PI who argued that his or her ideas had been stolen from a grant application by a reviewer.

All well and good. Unfortunately I tend to be a little suspicious when NIH staff pretend that all is perfect and well and the system works as it should and all that. You know how it goes. Things get testy. So here’s the rest of it. the walrus got all miffy and pouty and offered to take his or her toys and return home.

since you have such a low opinion of the abilities and integrity of NIH staff, I won’t waste anyone’s time here by offering any further responses to their questions.

HAHAHA. For the record, I do not, in fact, have a low opinion of the abilities and, most especially, the integrity of NIH staff.

What I said, reasonably clearly, is that I assume that there are very good reasons for a public face of NIH staff (SRO and PO) that is not entirely compatible with my accompanying assumptions that said staff are smarter than your average rutabaga. This has nothing to do with “integrity” and everything to do with professional requirements.

Now, if the walrus is telling me that his/her public/professional need to pretend that everything is always perfect and rose-smelling in grant review is the actual, entire belief, then yes, I’d have to revisit my assumptions in his/her particular case.

There is never any bias in review, reviewers are always perfect and expert and engaged, the review order is meaningful to the last digit, new investigators just need to “write better grants”, if Universities offer soft money jobs they “shouldn’t” be doing this and “why would you want to take a job there anyway”…..the list goes on and on for those things where the actual time it is on the street is not part of the official, public NIH line. I understand why they need to do this in some cases, not in others.

But not acknowledging that if you have lots and lots of scientists involved in reviewing grants there will be a suspicion of idea stealing at times? Ludicrous. And then going on to get all huffy just because I have one set of experiences and the walrus (and writedit) have another? hmm.

Okay, now on to the good writedit who seems to have let discomfort with my commentary overrun good sense.

Perhaps DM can restrict these sorts of discussions to his various for-profit plots in the blogosphere, where he is welcome to take on all comers.

My response: As you are very well aware, writedit, there is precisely one of those. Not plural, singular. As you are also very well aware, despite this scurrilous intimation, my tone and approach has been invariant on blogs that accrue “profit” to me (or any other entity) and otherwise. It is also the case that through the writedit blog you build additional credibility for what are most assuredly your primary professional talents and endeavors at present. Credibility that would, should you every see the need to deploy it, enhance your future job prospects. So if we are suggesting that someone is in this because they

get nothing out of maintaining this blog other than the satisfaction of being a good citizen and helping the biomedical research community through the exchange of useful information.

waayul, that is not strictly accurate in your case, now is it? I’d say this very real professional capital* that writedit has built amounts to a bit more than the beer money I make from the Scienceblogs.com DrugMonkey blog, wouldn’t you?

__
*don’t believe me? Which of you writedit (the blog) readers, if asked by your University if they should hire writedit (the blogger) to support their grant seeking faculty, would say anything other than “Yes dammit, right now!!”? Which of you would say that for some random administrator for whom you have no other evidence of their abilities? hmm? That’s professional capital.

More on the mega lab

April 11, 2011

Sally Rockey of OER posted data suggesting that most laboratories funded by the NIH are single-grant affairs. This was an attempt to deflate the belief on the part of many critics of the current state of NIH-funded affairs that, well, qaz put it best:

I think the people who complain about multi-grant labs/empires … are thinking “if only those multi-grant empires didn’t exist, there would be more for me, and, just maybe, I could survive.”

Remember, DearReader, that Jeremy Berg, Director of NIGMS, put up the following figure in a post on productivity per (direct cost) grant dollar?

Histograms showing the distributions of total annual direct costs, number of publications linked to those grants from 2007 to mid-2010 and average impact factor for the publication journals for 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006.

As we always note, it is unfortunate that this is data only from a single Institute. However, I still believe the null hypothesis has to be that these data are representative of the whole NIH until Rockey gives us more comprehensive data.

The point for today is in the left hand panel. If we assume the vast majority of R01 awards are at the modular limit ($250,000 / yr) or below, the histogram supports the contention on Rockey’s blog that the most populated category of investigators is that of those with only a single award. Account for those with a single award over the modular limit but under the special-circumstances* $500,000 limit, add on the second grant labs and it should be clear that the vast majority of PIs are captured.

So yes, numerically if you prevent the fifth, fourth or third R01 from going to a single PI, another investigator can be supported. But in terms of percentages, this is going to be a drop in the bucket. To get any significant traction you are going to have to go with the single-grant limit.

And that is just plain crazy talk.

Final Note: In all of this hoopla it continues to bemuse me that Early Stage and New Investigators are aligning themselves with the Benezra’s and St. Noonans. They are not your friends, people. They are the ones who have no problem with the holding pattern. They are the ones that think that having already been a NIH funded investigator counts more than the quality of the proposal or of the science. The ones that think that getting started off with NIH funding today is no harder then it was twenty or thirty years ago.

__
*Program has to agree in advance to take a R01 proposal that asks for $500,000 or more in direct costs per year

The question is posed by a new post from funkdoctorx:

The issue here is that the public does not understand what professors really do and how research works at the University level. Now, if you are a professor or postdoc reading this, think way back to the time when you were a first year graduate student. Remember how much there was to learn about the way the academic world, and academic research worked? Did you have much, if any, idea of this from your time as an undergraduate? I know I certainly didn’t grasp this at all. Even as a young post-doc I’m still working to understand how the system works despite being at it for 5+ years.

I was no different.

Read the rest of this entry »

One of the favorite “light the torches” solutions to the perceived problems with the NIH system at present is to limit the number of awards or the total amount of funding that one PI may hold.

Sally Rockey has a post up in which she rebuts the notion that this would solve anything.

I suspect that her data will stimulate a nice discussion! http://bit.ly/hwNJA8

December is one thing. It has to be expected value at this point that we in the NIH funded extramural research community are going to have to weather some weirdness for the December 1 funding date. To recap, grant applications submitted for the Feb-March receipt deadlines are reviewed in Jun-July, move on to Advisory Council (for a given IC) review in Aug-Sep and are generally proposed for Dec 1 for the first *possible* funding date. Whenever there is a failure of Congress to pass a budget authorizing the NIH to spend money for the new Fiscal Year, things get funky.

Typically, however, a budget has been passed with the NIH appropriation by the next “first possible funding date”, i.e., for grant applications submitted Jun-Jul, reviewed Oct-Nov for possible award as early as April 1.

You will have noted, Dear Reader, that we are past this date.

So I cruised over to RePORTER and found some 283 new R01s funded since 4/1/2011. This is in contrast with the 66 I found a few days into December 2010.

ICs seem to have resigned themselves to operating under this new reality. Either that or Newt Gingrich’s support has stiffened their spines.

The one thing we can’t tell from RePORTER is when these proposals were submitted. Could b leftovers from the prior round or could be from the new round. I’m betting they are mostly from the new round and the limbo-zone holdovers from December are still waiting. I’m sure the comments over at writedit’s place will clue us in on that.

Now, while I was a-RePORTERing, I happened to notice the pile of new grants funded by one of my ICs of greatest interest. One-third funded as A0, two-thirds as A1. Hmm. So I check another IC of interest – 50% on A1; another- 60%; another- only 7.5% on A1 (wowza)…and then just roughly looking at a bunch of other ones, I’d say the trend continues to improve for those grants funded on the first submission. We’ll want to keep our eyes on ones that continue to stink fund a lot of A1 proposals, of course. I think I may have mentioned before that I think two-thirds funded on the first submission and one-third on the A1 would be my initial target. If an IC wants to be serious about revision churning, that is.