Ignorance of the way review goes down is a dereliction of duty, NIH.
March 21, 2011
Grr.
On more than one occasion I have expressed confusion when the NIH reveals a bit of data or enacts a policy change based on something or other…and gives out the impression that the related emergent behavior of their systems has been a complete and utter mystery to them up to that point.
One pertinent example is the creation of the Early State Investigator category out of the previous omnibus New Investigator category. In each case it refers to someone who has never served as the PI of a major NIH research grant before. The idea was to make sure that newcomers to the system weren’t being totally blocked by the Good Old Boys and Girls who already enjoyed the NIH largess. I first served on a study section before they created the ESI category and I can tell you it took about half a day for me to realize what time it was. Namely that the “NI” apps that were most competitive were from highly established senior investigators who just didn’t happen to have previously needed NIH funding. Perhaps they were career NSF-grant folks. Or had made their careers in foreign institutions and had only recently moved to the US. But “New”, they most certainly were not.
The real “new” folks, the just-appointed Assistant Professors, were still taking in the teeth, of course. “If this was so readily apparent to me within a few hours of my first study section meeting“, I says to myself, I says, “surely the NIH knows all about this and intends for this to be the case“.
When the NIH started paying attention (see p 51 of this NIHPeerReviewReport) to those NIs that were only 10 years or less out from their doctoral award (i.e., the “ESI” designees), they expressed themselves as just having discovered this amazing fact about the previous success of different subsets of “new” investigators.
Then there’s the issue that was recently resolved (addressed? band-aided? concealed?) by elimination of the A2 revision. See p 32 of this NIHPeerReviewReport. The NIH needed this report to “discover” that study sections were responding to increasing applications and tightening funds by putting grant applications in a queue to “wait” for their chance at funding, via demanding revisions on the thinnest of grounds.
This is a fact I likewise discovered almost instantly upon joining a study section. It was a simple matter of seeing the sorts of things people were saying that led to original submissions being sent back for revision, phrases like “I’d like to see this one come back” uttered as if it was no big deal, applications on A2 being lauded despite being changed in no meaningful way from the A1, etc, and then doing a little judicious searching of CRISP (now RePORTER) by the study section to see the trend for grants that made it to funding.
Now Sally Rockey, who heads the Office of Extramural Research of the NIH, has a blog post up ( You Go Girl! ) describing some grant success data for women versus men.
Right now we’re celebrating Women’s History month. I recently was the keynote speaker at NIH’s observation of the month and was able to chat about my own scientific career and discuss research administration as a career choice. I was delighted to be asked to speak, as it made me reflect on how I came to be the head of extramural research at NIH through an unusual and circuitous route and to think about and celebrate the women who blazed the scientific trail to bring us where we are today. Thus, I thought I’d take this opportunity to provide you with some information from my talk regarding the participation of women in NIH extramural programs. These and other data on the topic are posted on the Web in the NIH Databook.
However, I’m curious to know whether anyone has examined success rates (gender related, etc) for grant applications reviewed by specific study sections.
Decent question. Smacks a little of personal whining but it really would be good to know if a study section or two had a big old gender issue, wouldn’t it? I mean, wouldn’t you want to fix that, NIH? All well and good to trumpet the general numbers but if a whole sub-area of Bunny Hopper women are having a dismal time of it, the general trends aren’t that helpful.
Some dim bulb wanted to know more about other topics:
You could start by breaking down the type 2s by the total project duration. If the difference is present at -06 this would be very worrisome indeed. An effect driven by the -16, -21 apps would be less than good but perhaps would mean it is wrapped up in generational issues.
What can you tell us about women’s success on the big mechanisms, Program Projects and Centers, in the PD role? And U-mechs?
Dr. Rockey’s answer is really telling.
These are interesting ideas. Unfortunately, we don’t have the data now.
It just boggles my mind that the NIH is so totally and completely unaware of the way study sections behave. The effect that their emergent cultures have on the selection of grants for funding. And how they seem totally ignorant that these emergent properties of review influence the state of their extramural workforce in detectable and significant ways. Ways that seem to be of eventual concern to them.
It is particularly offensive that an enterprise that exists to fund the collection and analysis of DATA has been so willfully ignorant of their own.
DrugMonkey 
March 21, 2011 at 4:19 pm
“Smacks a little of personal whining”
Maybe. I didn’t want to outwardly whine there, so I will do so here. I noticed from CRISP that the study section I kept getting assigned to did not have a single R01 grant awarded to a female. Not a single fucking one (at the time – I think there are now 2). There were a decent number of women who held R21’s and fellowship awards, but none held an R01. As a female nooB, I didn’t know if it was appropriate to bring this to the attention of the SRO – pegging me permanently as some whiny-assed beeyatch. All of my grant applications were triaged by this study section. Maybe they sucked. I don’t know. I eventually went to a different study section where I did receive a score, but never made it past the funding line. I was eventually funded by the NSF.
By the way — this:
“You could start by breaking down the type 2s by the total project duration. If the difference is present at -06 this would be very worrisome indeed. An effect driven by the -16, -21 apps would be less than good but perhaps would mean it is wrapped up in generational issues.”
I have no idea wtf you are talking about. What’s a 16 or 21 app? Whatsa type 2???
LikeLike
March 21, 2011 at 4:35 pm
The idea of NSF funding sociological research (e.g. $DAUGHTER) into NIH funding mechanisms somehow strikes me as the kind of thing to cause knives to be sharpened in Washington.
LikeLike
March 21, 2011 at 4:50 pm
It reminds me of don’t ask and don’t tell. They don’t know because they don’t want to know, so they can have plausible deniability and not feel “responsible” for the havoc that they have caused to happen.
It reminds me of a science fiction story where the problem of overpopulation was so great that the world’s “leaders” contract with an alien race to eliminate slightly more than half the human population. The “leaders” don’t want to know the details because it would be unethical to “play God” and determine who lives and dies. They delegate that to the aliens. So the aliens kill all the human females. Overpopulation problem solved.
LikeLike
March 21, 2011 at 5:01 pm
And what is the graduation rate, as well as publication rate and job placement rate, for the graduate programs you are involved in? Or heck, just the statistics for your own trainees, and those in your department?
If you can answer accurately, with appropriate numbers, and anyone else can access that data, bully for you. And please tell me so that some of my extraordinarily frail faith in humanity might be restored.
Frankly, the NIH is a big enough monstrosity I’m not at all surprised they haven’t started doing this yet. The fact that my university doesn’t… that disturbs me a bit more.
LikeLike
March 21, 2011 at 8:14 pm
hi anon from rock blog, welcome if you are new to the DM blog…
yeah, that sounds pretty damning for the study section you are talking about. definitely worth bringing to someone’s attention, especially now that you have escaped the section? maybe you can get somewhere appropriately anonymous and drop the bomb on the OER blog… 🙂
Type 2s are competing continuation applications. So the numbers refer to the year or continuation, assuming each interval was a 5 year proposal. That would make the first competing revision named the bladyblah-06 for the 6th year of funding. the third continuation would be -16, the fourth -21. My point was that if this was being driven by OldOldeBoyes who have been in the game for 20 years, that’s different than if the effect is present as early as the very first attempt to continue the grant.
LikeLike
March 22, 2011 at 8:59 am
DM – thank you. I’ve been here before and post stuff here and there. I enjoy this blog and some others.
becca – I see your point. But the NIH has all of this information in their electronic database. I myself can gather info about the successfully funded projects per study section (via Reporter [which, by the way, is a stupid name]), but have no access to the unsuccessful proposals. That they claim to not have this data is bullshit. Physioprof has blogged about “knowing” your study section well. It may have been helpful to me to have known ahead of time some of their non-PR behavior.
LikeLike
March 22, 2011 at 2:25 pm
It is particularly offensive that an enterprise that exists to fund the collection and analysis of DATA has been so willfully ignorant of their own.
Nicely put. I agree completely. But this seems to be the modus operandi for NIH in general.
For example, these were my exact thoughts when I learned that NIH didn’t know how many postdocs there were, or where they ended up.
But my experience at a so-called diversity workshop was by far the most appalling. The emphasis seemed to be entirely on minority discrimination, so I asked why we weren’t discussing gender discrimination at all.
I was told that “it’s not a problem anymore”.
I said that was not my experience.
The woman in charge was clearly offended and upset that I would suggest such a thing. She said the data show that “it’s better now”. I said maybe, but not better enough, and they haven’t been collecting the right kind of data in more than ten years, or it would be obvious.
I was so mad, I didn’t care if I pissed her off. I was furious that a senior woman at NIH willfully refused to consider the possibility that the elephant’s phallus is still in the room.
Of course, in retrospect I should have known this is how NIH operates. Like an ostrich wiggling its feet and wondering why it can’t go anywhere with its head buried under the sand.
LikeLike
March 22, 2011 at 3:48 pm
The NIH is also intensely ignorant of any kind of metric of its own successes, or lack thereof, indicative of a generalized lack of introspective analysis. NASA can say, “we flew these missions, here’s the pictures.” The military can say, “here’s the bombers we ordered and here’s the stuff they blew up.” The NIH has got nothing other than a lot of insubstantial hard-to-quantify feel good about itself. This is a problem because the NIH has difficulty justifying its budget to Congress, which in large part explains why the NIH budget has been flat/decreasing in real terms for the last 8 years. To me, this is the driving force behind Collins’ rebranding/transmogrification of NCRR into a “translational” institute: he wants something to show the public — we invested $X and these are the drugs we got and over here are the taxpayers those drugs helped.
LikeLike
March 23, 2011 at 10:08 am
NASA can say, “we flew these missions, here’s the pictures.” The military can say, “here’s the bombers we ordered and here’s the stuff they blew up.”
The trouble is that the timescale for basic (biomedical)research to pay off in practical terms is 20-30 years. We should be reminding people of all of the drug/treatments that are saving or improving lives now that wouldn’t exist if the groundwork wasn’t laid during the 80’s-90’s. That should be an easy case to make because drug companies are generally pretty open about admitting that they rely on NIH funded basic research.
The worst thing to do would be to get into the business of saying “Double our budget and we’ll give you miracle cures in 5 years!”
If you want to kill public support for NIH research, I can’t think of a better way of doing it than that.
LikeLike