GrantRant XI

January 31, 2013

Combative responses to prior review are an exceptionally stupid thing to write. Even if you are right on the merits.

Your grant has been sunk in one page you poor, poor fool.

Advertisements

The glass is half full

January 31, 2013

There is nothing like a round of study section to make you wish you were the Boss of ALL the Science.

There is just soooo much incredible science being proposed. From noob to grey beard the PIs are coming up with really interesting and highly significant proposals. We’d learn a lot from all of them.

Obviously, it is the stuff that interests me that should fund. That stuff those other reviewers liked we can do without!

Sometimes I just want to blast the good ones with the NGA gun and be done.


Notice of Grant Award

GrantRant X

January 30, 2013

Open the grant you are polishing up right now, pronto, and change your reference style to Author-Date from that godawful numbered citation style. Then go on a slash and burn mission to make the length requirement. Because lets face it we all know your excuses about reading “flow” are bogus and you are just shoe horning in more text.

 

That numbered citation stuff is maddening to reviewers.

 

happy editing,

Uncle DM, Your Grant Fairy.

grants_per_pi_allThis figure was posted by Sally Rockey, head of the Office of extramural research on her blog Rock Talking. We have discussed these data in the past but given my recent Fixing the NIH series of posts, I thought it worth bringing up.This depicts the number of investigators funded by the NIH who hold a given number of Research Project Grants as Principal Investigator. This includes a range of R-mechs, U-mechs, DPs, Program Projects, etc. As Dr. Rockey noted:

If you crunch the numbers, you will see that in each of the four years presented more than 90 percent of our investigators hold one or two research project grants.

And, if as I do, you spy a slight trend for increasing numbers of grants per investigator over time, I refer you to my analysis of the real purchasing power of the full modular ($250K in direct cost per year) R01. The short version is that the full-modular of FY 2011 had 69% of the purchasing power of the same award in FY 2001. A PI needs about $350K per year to have the same grant. Also note that the chances of suffering budget reductions upon funding and even on non-competing renewal, due to ongoing Continuing Resolution problems with Congress, has increased relative to the early noughties. So a little aggregate creep in the number of grants per PI should be expected.

The reason for Dr. Rockey presenting these data in the first place coincides with my reason for posting the graph today. Because one of the very popular “fixes” for the NIH incorporates some version of the assertion that there are some PIs who enjoy a huge amount of funding and that by preventing them from doing so, we’ll be able to give a lot more people a basic level of support. “Blood from a stone” is not precisely the right aphorism here but suffice it to say, there aren’t enough of these hugely funded labs to make a difference. Yes, of course, one R01 subtracted from Professor MoneyBags could go to Asst. Professor J.R. Mint and thereby make a HUGE difference in her life. But on the order of systematic fixes…..this does little.

9 types of PIsThis is the now famous cartoon by Dent which describes “types” of Principal Investigators. Obviously it is drawn from the perspective of trainees (no?) but it resonates with a timeless truthiness. To me anyway. And it is somewhat useful in our considerations for how to fix the NIH. We have to reduce the number of mouths at the trough, this is obvious. I’ve proposed that we need a prospective approach for the medium to long term future. I here renew my assertion that we need to get specific about which type of PI is to be put in the gunsights for reduction.

Obviously, the answer is “That guy! over there….yeah, HIM. Not me, nuh-uh, I need to be preserved at all costs, dude!

Sigh.

This is precisely why we need to have this conversation and precisely why the NIH needs to get more serious about making this hard call for themselves. Otherwise the culling will continue in an uncontrolled, random bolt-of-lightning fashion. Rockey called this “Darwinian” in (I think) a quote in a Science news bit AAAS bit. I don’t think that is quite the right term….laissez-faire maybe? At any rate. They are going to have to make the culling intentional if they have any interest whatsoever in 1) quality differences between funded investigators and 2) ensuring that the pool that remain after this great culling occurs is as high in quality* as possible.

One possible axis for pursuing this more-rational culling of the herd should involve thinking about types of operations. Small town grocer? Glamour Hound? Dreamer? Slave driver? Are any of these PI phenotypes associated with better value received for dollar spent by the NIH? Or is PI quality entirely uncorrelated with “type” of operation?

Raise your eyes back to the first graph. NIH has a lot of information on PI “type” based on the number of grants / dollars awarded. Some additional relevant information from University or Institute “type”. Does the size of the total NIH extramural portfolio (dollars, numbers of PIs, etc) at local institution influence success? They can, if they choose, do a bunch of retrospective peeking along a given PIs career track to see if a certain funding threshold at various points in the career are associated with success / failure. They need these data to evaluate Michael Eisen’s thresholds, btw.

I plead with you. As you engage in this discussion around and about…on blogs and in real life conversations….try to focus on the data we have. And the analyses that the NIH could conduct in the future. For these latter, demand them over at Rock Talk. Try to temper your knee-jerk “do it to that guy over there, my type of investigator is the BESTEVAH!!!” with some consideration of a larger picture. It is HARD. Believe me I know. Like I said elsewhere, I have no desire to be culled. None whatsoever. So obviously I’m looking very hard for arguments for why my type of scientist, my type of science, my type of job category and my type of institution are providing the best value. Given this, we should all double down on understanding and integrating data if it is available.

Recognizing that some 90 percent of PIs in the NIH system have only one or two RPGs is an example of what I mean.

Additional Reading on Fixing the NIH:
The NIH must dismantle the corrosive competitive culture of science
Shut off the PhD tap
We are going to fix the NIH

__
*yes there are many qualia that could be of interest here.

I like competition, don’t get me wrong. I engaged in inter-school competitive sports from freshman year of high school through my senior year of college. I played intramural sports from late high school through the end of graduate school. I’ve done competitive sports outside of school organizations from high school until…yesterday. Essentially uninterrupted.

Nowadays, I spend a solid plurality of my weekends schlepping one kid or another around to a competitive sporting event.

roids_mcgwire.jpg
Just milk? source
I love what competition does for us on many levels, of course. This should be obvious from the above. Of the many benefits, one thing competition does that is most useful is to make us strive to be better. It makes us practice to improve our play and our game. It makes us get fitter, more accomplished, more capable. It makes us attain performance levels we didn’t know we could reach.

This is true in science as well.

Science is indeed a competitive business, as most of my Readers know full well.

We positively reify the markers of success- getting a particular scientific discovery first. Accomplishing some demonstration or discovery with the greatest panache. Coming to a realization or theory that changes the way everyone else thinks about a topic. Creating a medical therapeutic approach…or the basis for such a thing. The accolades are both arbitrary (prizes, “respect”) and specific (grant funding, jobs of increasing worth, etc). At heart, scientists are trying to learn things about the function of the natural world and so there is an overlying competition to advance knowledge.

In all of this, the pot is sweetened by the competition. The scientist receives part of her respect not merely for accomplishing a certain task but for doing it before, or better than, the next scientist.

AP photo from here

This reality can be fantastic for science. As in sport, the competition makes us work harder, make us work to up our game and motivates our excellence. This speeds the advance of knowledge. One of my favorite formative anecdotes was the late 80s-mid 90s competition between several laboratories to comprehensively identify the role that various medial temporal lobe structures (e.g., the hippocampus) played in memory function. In this case the competition was made more acute (as it often is in science) by disagreement. One lab thought structure X really did Y and another insisted it did Z. Or Y’, perhaps. And every year the Society for Neuroscience annual meeting would have a hilarious slide session in which the labs would bash away at each other. Almost always…pointedly. Sometimes in semi-personal attacks. Then they would scurry back to their labs, publish a paper or three and come back next year ready for more battle with their latest results. Understanding was advanced.

This is where we depart from competitive sports.

The key feature in my anecdote is that the labs would publish. Most if not all of their results. And they would discuss their latest findings at meetings. With. their. competitors. Knowledge was built not just by the major players but by anyone else who cared to chip in as well. Because there was a superseding goal that went beyond the simple question of who crossed a line first or who scored the most points.

That goal was the provision of knowledge to everyone. Because scientific advance requires a collaboration amongst many. This is why we publish papers that include full methodological description. This is why we are expected to be honest about how we did a particular study. This is why we are expected to share the very intellectual property that was necessary for the experiments!

People seemed to understand this, and acted accordingly, during the medial temporal lobe memory warz.

The trouble comes when we start behaving a little too much like sports competition.

Before I get into it, another analogy. Take business. It used to be that competition was about money, yes, but also about providing a service or building a widget. Making something that people wanted and needed. The marker of success was not just driving your competitors out of business…but in being the best to provide rail service from New York to San Francisco. To supply an automobile that people could afford….and that worked. At some point, business became more about scoring the most points or crossing the tape first. The role of arbitrary performance indicators (unimaginable sums of money, unconnected to anything that could be viewed as necessary for the participants) in motivating behavior totally supplanted real indicators. And in many cases the product or service suffered tremendous harm. As did the consumer.

We have reached this point of transition in science. The marker of success is the mere fact of publication* of a paper in Journals of established, but arbitrary, rank. It is no longer about the actual finding or any sense of advancing science or knowledge. Papers are increasingly disconnected from each other and from anything that is of any reasonable importance to know.

So why should the NIH care? No, I don’t mean for the last point here. Yes, the relevance of work funded by the National Institutes of Health does concern me. However, the appropriate valuation across the scales of “basic” to “applied” research are not the topic of today.

The topic of today is the efficiency with which the science that the NIH pays for is advanced.

Sadly, we are in a time of great secrecy within science. Because being first** to some finding is rewarded above and beyond all other things, the very essence of the competition demands not letting anyone else know what you are doing until it is published. The typical manuscript in our most respected journals requires many person-years of work. And much of this work never sees the light of day for various reasons. It is negative. Merely supportive. A blind alley. Or perhaps just of insufficiently amazing interest.

More sordidly, much of this work never sees the light of day because it might help a competitor lab to beat us next time.

This is being done on the NIH dime. Right now, in labs all across the US. Many, many hours and $$$ of work being conducted that will never see the light of day (i.e., be published).

Admittedly there is a lot of work that nobody wants to see. I get this. I am no fan of Open Notebook Science. I want scientists to present their work to me somewhat triaged for interest. But we are well down the road from that level at present.

The “cost” to the NIH is not merely the invisibility of data and findings that they have already paid for. It is also in the future expenditures as another laboratory has to repeat the same experiments, generate the same blind alleys, waste the same time evaluating bad reagents or theories.

Sadly, some labs even lay a false trail by describing their Methods so incompletely that other labs get a wrong impression of what needs to be done.

This can burn years of a trainees time in a lab. No joke and no exaggeration.

And we haven’t even arrived at the discussion of fraud which is also driven by the arbitrary markers of competition.

Time for the NIH to get interested in the way that competition for arbitrary markers in science is wasting their precious taxpayer dollars. Long past time. I’m thinking of writing Sen Grassley’s committee myself! (kidding.)

Solutions? Well, we’re faced in part with a Justice Potter Stewart solution in that we can identify wasteful, GlamourPublication chasing laboratory operations when we see them. We can also take a stab at estimating how many person hours of work are surely being buried in the process but this will start to get a little…forensic. But if it were easy…..

I’m going to suggest going after Glamour idiocy at two places. Empower the Program Officers to demand a better ratio of work payed for to publications resulting, first. Second, the study section. Yep, beef up the analysis of “productivity” by creating a set of bullet point guidelines for how to asses. They have them for the other aspects of grant review, right? The Significance, Innovation, etc criteria? Well, no problem beefing up the assessment of Productivity.

Heck, this should be a formal criterion on all grant review, not just continuation proposals. It dovetails nicely with Michael Eisen’s proposals for lab-based or person-based funding, doesn’t it? How many people have you had working in your lab and how many figures have been published? What is your total lab support, including fellowships, TAships, etc for your trainees? Have you published as much of this work as you possibly can?

Or are you engaging in competition for arbitrary markers and are relegating much of the work to the dark corners of forgotten harddrives?

Additional Reading on Fixing the NIH:
Shut off the PhD tap
We are going to fix the NIH

__
*If you ever catch yourself saying “my Cell paper”, “the Jones lab’s Nature paper” or “her Science paper” in preference or addition to a short description of the topic of the paper….you are part of the problem. And you need to step back from the brink of GlamourDouchery before you fall in for good.

**Two labs could have the essentially same idea about solving a given problem, say the function of a gene. They could beaver away with 5-20 people contributing various science over the course of years. With many millions of dollars of NIH funds expended. If they happen to wrap up their “stories” a mere two months apart, this can be the difference between being accepted into Science or Nature or not. It may even be the case that the second one to be ready is a better demonstration on all features and yet the priority, the mere fact of submitting it for consideration first, rules the day. This is profoundly disturbed.

What is even more disturbed in the system is what happens next. Many aspects of the paper which has been beaten to the punch may not be published at all! That’s right. For the type of lab that is competing on the “get”, i.e., the mere fact of a Nature or Science acceptance, it is “back to work, minions!” time. Time to take the “story” beyond the current state of affairs and hope to win the priority battle for the next story which is big enough for Science or Nature to take it. At the very least replication is lost. More likely, there are a number of differences between the two studies, differences that maybe were of interest to other laboratories. Of interest for different reasons to the same laboratories. Or may later come into focus ten years later because of additional findings. Yet because of the competitive conundrum of science, many of those findings will be lost forever.

ETA: Forgot my disclaimer. I have, in many ways, tried to run to daylight in my scientific choices. This is in part due to what is an intrinsic orientation of mine, in part due to accidents of training history and in part due to explicit decision-making vis a vis the career on my part. I avoid competitive nonsense. I am not in the Glamour Chase. I am not entirely certain whether or not various steps to dismantle the bad effects of Glamour chasing, scooping, priority focused science would be good or bad for me, to be honest.

Shut off the PhD tap

January 28, 2013

We need to stop training so many PhD scientists.

It is overwhelmingly clear that much of the quotidian difficulty vis a vis grant funding is that we have too many mouths at the NIH grant trough. The career progression for PhDs in biomedicine has experienced a long and steady process of delay, impediment, uncertainty and disgruntlement, things have only gotten worse since this appeared in Science in 2002.

The panel’s co-chair, biologist Torsten Wiesel of Rockefeller University in New York City, is surprised to learn that this aging trend continues today: “You’d think with all the money that’s going into NIH, [young scientists] would be doing better.” His co-chair, biologist Shirley Tilghman, now president of Princeton University, says simply, “It’s appalling.” The data reviewed by the panel in 1994 looked “bad,” she says, “but compared to today, they actually look pretty good.” She adds: “The notion that our field right now has such a tiny percentage of people under the age of 35 initiating research … is very unhealthy and very worrisome.” …Experts differ on why older biomedical researchers are receiving a growing share of the pie these days and on what should be done about it. But they agree on the basic problem: The system is taking longer to launch young biologists.

We need to turn off the tap. Stop training so many PhDs.

NDBSlide243_012813This is going to hurt the many, many of us (and therefore the NIH) who depend on the undervalued labor of graduate students. This chart (click to enlarge it) from the NIH RePORTER site shows the relatively slow increase in NIH funded fellowships and traineeships compared with the more rapid increase in research assistantships (light blue). Read: graduate students paid directly from research grants. The more graduate students we “train” in this way, the more we need to secure more R01s and other R-mech grants to support them.

Spare me your anecdotes about how graduate students cost as much as postdocs or technicians (to your NIH R-mechanism or equivalent research grants). If they weren’t good value, you’d switch over. The system, as a whole, is most certainly finding value in exploiting the labor of graduate students on the promise of a career that is now uncertain to be realized. This is because the charging of tuition and fees is still incomplete. Because students have the possibility at some point during the tenure in our laboratories of landing supporting fellowships of various kinds. Because some departments still receive substantial Teaching Assistant funds to support graduate students (and simultaneously ease the work of allegedly professing Professors). And above all else, because we are able to pull off an exploitative culture in which graduate students are induced to work crazy hard in a Hunger Games style bloodthirsty competition for the prize….and Assistant Professor appointment.

It is going to hurt undergraduates who may wish to become PhDs and now cannot compete successfully for an admission to what are, presumably, going to become increasingly selective programs. I regret this. I am a huge fan of the democracy of our academic system and I wish to let all who have an interest…try. I have come to the belief that at this particular juncture, the costs are simply too high. The ratio of those who enter in pursuit of a particular outcome (Professordom) to those who achieve it is just too low. We need to rebalance. Part of the pain will fall on the undergraduate who wishes a career in science. Their chance to compete will be abrogated.

This is, in the short term, going to hurt the NIH’s output per grant dollar. Across the board, this labor is going to have to be replaced with research technicians*. People who get regular raises, benefits and work a more traditional number of hours per week.

But it will shrink the balloon of PhD trained people who are hankering to get into the NIH system as, eventually, grant-funded PIs. This will be a good thing in the end.

UPDATE 01/29/13: Check this out!

via ChemJobber
__
My honest disclosure is that this one is painless for past me and current me. First, I was a fairly decent candidate for graduate school when I applied. I looked good on paper, etc. I assume that I would still have been competitive for at least one of the four offers I received out of five applications. Second, I have made my way as an investigator without much reliance on graduate students labor. So for me, this one is painless. Shutting off the tap of graduate trainees wouldn’t have changed the way I have done research up to this point.

*One likely outcome is that graduate training and postdoctoral training is going to have to include more managerial approaches. Yes, this happens spottily across all of bioscience at present but as a population, it will increase. It will involve more supervision of techs earlier in the PhD training arc. I think this is a good thing.

As noted recently by Bashir, the NIH response to the Ginther report contrasts with their response to certain other issues of grant disparity:

I want to contrast this with NIH actions regarding other issues. In that same blog post I linked there is also discussion of the ongoing early career investigator issues. Here is a selection of some of the actions directed towards that problem.

NIH plans to increase the funding of awards that encourage independence like the K99/R00 and early independence awards, and increase the initial postdoctoral researcher stipend.

In the past NIH has also taken actions in modifying how grants are awarded. The whole Early Stage Investigator designation is part of that. Grant pickups, etc.

I don’t want to get all Kanye (“NIH doesn’t care about black researchers”), but priorities, be they individual or institutional, really come though not in talk but actions. Now, I don’t have any special knowledge about the source or solution to the racial disparity. But the NIH response here seems more along the lines of adequate than overwhelming.

In writing another post, I ran across this 2002 bit in Science. This part stands out:

It’s not because the peer-review system is biased against younger people, Tilghman argues. When her NRC panel looked into this, she says, “we could find no data at all [supporting the idea] that young people are being discriminated against.”

Although I might take issue with what data they chose to examine and the difficulty of proving “discrimination” in a subjective process like grant review, the point at hand is larger. The NIH had a panel which could find no evidence of discrimination and they nevertheless went straight to work picking up New Investigator grants out of the order of review to guarantee an equal outcome!

Interesting, this is.