Some NIH grants should fail

We have another version of bash-the-R21 brewing, for previous work from PhysioProf on the topic see here, here and here.
The discussion ended up touching on the paralytic meme that it is impossible to get an R01 funded without copious preliminary data testifying specifically and empirically that a large part of the proposal is/will be supported.

It doesn’t help me to say, “You should go for the R01” when I have what I think are great ideas, prelim data to show the ideas are feasible, but not enough to justify an R01 or defend against “fishing expedition” criticisms. Not to mention a publication track record. I don’t resent this — if I was giving a PI $500K I’d give it to the PI who has years of great publications and boatloads of preliminary data too.
But this is why I (in a basic science dept) am primarily applying for NSF and R21s for now, hopefully in 1-3 years I’ll have the data for the R01. Is this wrong?

See what I mean about “paralytic”?

The commenter, MBench, goes on to ask:
Should I be floating out an R01 just in case I get the review panel that decides my project is terrific despite my few-pubs-as-an-independent-PI-and-bare-bones-prelim-data?

Three out of three Internet blovinards agree…
pinus: yes
whimple: yes
YHN: yes
…but musing on this topic returns me to one of my usual themes which is the underlying reason why the NIH/CSR review panels are so fixated on Preliminary Data. The best I can deduce is that it derives from a belief that part of the job of review is to weed out those projects which will “fail” from those which will “succeed”. And this is pervasive, let me quickly acknowledge. It is frequently at the root of the review comments about investigator juniority/seniority, environmental support, past-productivity and the like.
Sometimes the concept of failure is pretty explicit- as in “That experimental design will never provide any clear evidence bearing on the hypothesis”. or “That person, in that environment, cannot possible build, validate and deploy that research capacity in the requested time for the requested money”. But even where the criticism is reasonably explicit, it should be clear to us all that it is merely an empirical prediction. Sometimes the criticism comes with good supporting rationale, but sometimes it is supported only with lazy and unthinking rationale. Personally, I urge reviewers to do the best they can to focus on exactly what it is that they are predicting. To be clear about whether they are confusing their experimental-outcome predictions with an evaluation of whether a good empirical test of various experimental outcome predictions will be conducted. The latter is an appropriate area of concern, the former is far less frequently a valuable critique.

If you aren’t crashing, you aren’t trying
(source)

Nevertheless, one thing that I think has an insufficiently prominent role at the study section table is the consideration that failure can be a good thing! We rarely even consider what the notion of failure means. There are various failure modes in the NIH funded grant game, of course. Failure can mean that a junior investigator is awarded an R01 for 5 years, has trouble building a research program, the studies come out to crap, s/he never publishes any data and never seeks any subsequent research funding. Does that cover more or less a complete and utter failure of a NIH grant award? No data, no pubs, no subsequent awards based on promising work and no career. Pretty much a zero gain.
We move up from there. Perhaps parts of this scenario succeed and some others fail. Maybe it is only that the papers are mostly negative. Or there are papers but they are only methodological works because the intended investigations mostly turn out crappy, disappointing, frustrating or whatever. Or perhaps the award goes swimmingly with lots of specific pubs… but the investigator is subsequently lost forever to NIH-funded and/or public health related science. Sure there are some gains associated with the award but it is not a trifecta of papers, specific tests of the hypotheses and a continued productive research program.
Feel free to contrast this with some versions of the successful NIH grant award. A steady stream of publications which move, robotically, through a series of Specific Aims as originally proposed. Subsequent research awards which build on the work already completed. More awards, more papers and more Specific Aims.
Aims which are duplicated, more or less, in three other labs around the country working on nearly indistinguishable projects. All of them reference and cite each other, djinning up some decent cites and buzz. (Let us not be too snarky, we can admit that the general area is important to the mission of the NIH and to science, it is just that there is nothing uniquely awesome about any of the participating labs work and hypotheses…) The investigator goes on to acquire other awards, renews this award and generally becomes just another one of the gang. Is this a “success”? Well, yes it is. Science is indeed incremental- it is hard to predict what is going to later be seen as critical groundwork. Even the most similar of laboratories come up with a shining unique gem, now and then. A gem that may distinguish their program forevermore going forward.
But there is that little nagging problem of opportunity cost. What has been overlooked in the conservative rush to fund safe, unrisky me-too proposals that are well supported by the Preliminary Data?
The glorious? The paradigm shifting? The quantum leap in understanding perhaps?
To be honest, I think at the core of their being, many reviewers do recognize the need for a diversity of scientific investigations, from the safer/pedestrian to the risky/high reward. When it comes to actually reviewing grants, however, I think the nebulous concept of a grant success is given too great of a role.
If there is an NIH standard(ish) mechanism which begs for failed grants, the R21 is it. I wonder how many of them succeed and how many really turf out? How many lead to one great paper, how many lead to at least one subsequent grant award and how many lead to a big fat goose egg?
Remember NIH, if you aren’t crashing some of the time then you aren’t really trying to get better.