On recruiting peers to review NIH grants
June 30, 2015
May 2015 Advisory Council round for the CSR of NIH.
I’ll be making observations on the Luci Roberts presentation in a little while. For now, enjoy.
UPDATE:
Okay, down to business. The part I wanted to highlight starts at 1:35 of the videocast. Dr. Nakamura introduces Luci Roberts, Director of Planning and Evaluation in the OER.
A comment from baltogirl
It is a little known fact that SROs have trouble recruiting for study section (I was told that two-thirds of people asked decline to serve). It’s likely that most people are so busy writing their own grants they can’t take a full month off to devote to reading the work of others.
reminded me I forgot to discuss these data.
The Division of Planning and Evaluation conducted a survey** some time ago, trying to determine the attitudes of extramural PIs that would affect their willingness to serve on study sections. I thought it applied to the above comment.
They surveyed 4,000 individuals who had submitted at least one grant as PI in the past five years and who had obtained active funding (any variety) from the NIH in the past five years. 1830 or 46% responded. Not too shabby. (They also surveyed 423 SROs of which 271/64% responded, more on that later.) Of the PIs 1,616 had served as PIs or PDs, the balance were TG directors or subproject/consortium PIs.
First up. 964 (53%) had never been asked to serve on study section in the 12 months prior to the survey. Hmm. I can almost stop right here. But no….there’s much more. Still, if there is a reviewer crunch, the first order of business is to determine why over half the potential pool is not even being asked.
Ok, of the remaining 866 individuals who were asked to serve on study section, 762 agreed to do so. That’s 88% saying yes. So the rumor that “two-thirds of people asked decline to serve” is falsified by this survey. Clearly, the vast majority of people who are asked step up and do their community duty.
I really, really like this. It is heartening.
The next most-interesting thing was the 10th slide which shows the ranks of PIs who are asked/not asked to review. As you would expect, the Full/Associate/Assistant Professor ranks for those asked ran 54%/32%/11% (that high for Assistants?) and 22%/23%/37% for those not asked to serve.
Again, this outcome makes it really clear what needs to be done if getting reviewers is a problem. Ask more Assistant Professors to serve***. Right? Done.
Then we get into a couple of slides on why people might say “no” when asked to review. Slide 11 present the top reasons (out of an open text box response, per Roberts’ presentation) for the process being “more burdensome than it could be”. The numbers are confused here because the denominator appears to be 861 when it should be 762. But in any case, 630 of the respondents who reviewed said the process was not more burdensome than necessary (huh? surprised on this one). Of the 158 who said it was too burdensome, 45% complained about the number of assigned applications. The next most common (16%) complaint was “too much time devoted to applications that will never fund”. So pretty much, the most burdensome thing was review load.
This brings us to Slide 13 which pits SRO opinion versus how reviewers think. One of the biggest disconnects was in the number of in-person meetings per year that is “reasonable” since reviewers lean 1-2 and SROs were about evenly split between 1-2 and 3-4 as okay. A similar disconnect was found on application load. Half of the reviewers felt 4-6 apps was a reasonable load and only 25% felt 7-9 was reasonable. SROs leaned 7-9 (~60% of SROs) with less than 40% finding a 4-6 grant load reasonable.
Predictably.
I’m skipping Slide 12 on reviewer and SRO thoughts on reason to accept and decline invitations to review since 88% of those asked say yes anyway. Who cares what they say if most of them will do it just for the asking?
Okay, those were the things that jumped out at me.
___
*As a disclosure, Dr. Roberts was once a SRO who played a highly formative role in the early-career understanding of the NIH grant business for YHN. She was kind enough to send me the slide deck that was used in her presentation.
**I was a respondent, fwiw.
***In case of any newcomers, both YHN and CPP have advocated for this on this blog since forever.
Question of the day
June 29, 2015
What percentage of K99 should fail to transition to the R00 phase in a healthy system?
What percentage of those that go to R00 should fail to ever gain major independent funding as a PI?
PhysioProffe on the conduct of science
June 25, 2015
Self-interested nepotistic shittebagges constantly assert this parade of horribles that if we don’t fund the right subset of scientists in today’s tight scientific funding environment (coincidentally them, their friends, their trainees, and their family members), then we are going to destroy scientific progress. This is because they are delusional……
Peer review and the death sentence
June 25, 2015
It is relatively easy to kill grant applications because the reviewer knows the applicant can always apply again.
Same thing for hiring decisions because surely some lesser University will hire the three other candidates on the short list.
In many tenure cases, the Department knows that this person will get a professorial rank job elsewhere*.
Germain’s scheme is going to require peers in the field to pass a death sentence on the career. And to make the numbers add up, there will be a LOT of this.
Those peers know that they themselves will be up for chopping in the next 5-7 years.
I do not foresee much enthusiasm for scoring progress as deficient and this will only grow more intense with each successive 5 year review interval.
__
*and anyway these are so personal at this point that it is a different matter.
Republicans of Science
June 25, 2015
Predicting the future
June 24, 2015
One of the biggest whoppers told by Ronald Germain in his manifesto on fixing the NIH is this:
it is widely accepted that past performance, not a detailed research plan, is the best predictor of future success. So why stay with the fiction that R01 grant proposals are the best method for determining support of the individual scientist,
As I often say there is nothing so a-scientific and illogical as a scientist on the business of science.
The way he states this plays the usual sleight of hand with the all-important, unmentioned variable.
Namely, the means to do the research. Grant funding.
There is no bigger predictor for the success of a given research plan submitted to the NIH than whether or not the PI receives the funding to do the work. Funnily enough, Germain actually recognizes this and totally undercuts his argument in one of his caveats:
with 5–7 years of support per round and 1–2 years of bridge funding available, I think it is unlikely that a highly competent investigator will fail to produce enough during 6–9 years of research to warrant a “passing grade” without further extensions, except in extenuating circumstances.
Right? He sees right here that all that matters is funding. Most competent investigators will succeed if they but have the funding! Which makes his idea that this will cut down on competition and the “stochastic” nature of getting the grant funding look as silly as it is.
It’s just another way to say “We’ll pick our favored winners in advance of any independent accomplishment based on who they trained with (i.e., us!) They will keep right on winning because they will be the only ones with the means to accomplish anything. All others can stay the heck away from our effortless stream of moola, no matter how good their ideas might be“.
This is important and it is why basing funding on accomplishment, rather than great ideas and the capacity to fulfill them is recipe for a death spiral of the Extramural NIH productivity as a whole.
This plan will self-reinforce and harden a silo around a limited set of brains, doing science in the way they see fit. Good ideas from outside this silo will not be given a chance to compete….unless they happen to occur to someone inside the silo. And on the whole, that person will not represent a diversity of ideas, approaches and interests. This will, across the enterprise of NIH-funded science, reduce the rate of discovery.
Those who manage to accomplish will continue to have a stranglehold on the means to accomplish. Means leads to accomplishment leads to more means in the Germain scheme.
So what gets accomplished will be narrowed, iteratively, with each 5-7 year review. Only to be refreshed, minimally, with each squeezed down cohort of new hires who manage to make it into his starter, block-grant scenario. Those, of course, will be selected by Universities on the basis of seeming like the people who are already most successful since the review will be anticipated to be on the basis of the person. Naturally, the trainees of the insider club will be most highly sought after. (Take a look at the way HHMIers, espcially the Early Career ones have been trained folks. …talk about the past predicting the future and all, right?)
So when you hear someone talking about “the best predictor of future performance is past performance”, make sure to ask whether that is with or without the funding and how they know this.
The second truthy whopper Germain tells follows soon after.
true creativity is often cause for lower scores?
Personally I have yet to see a well-prepared truly creative grant get killed just for being creative and new. Maybe wackaloon geniuses who have great ideas but simply refuse to write an actual grant proposal struggle in some sections. I guess. But here’s a secret for Germain. (A “secret” known to just about anyone who has served on 2-3 traditional standing study sections.) People that he is talking about, those who have demonstrated a high level of accomplishment in the past 5-7 years, get away with utterly crappy proposals and still get their funding based on their record of accomplishment.
That’s right. We ALREADY have a system in place that HUGELY benefits and prioritizes the funding of people with a track record of accomplishment. The “creativity” in their proposal does not prevent them from getting funded. Nor, btw, does diverging substantially from the plan they got the money for hurt them in the next round of evaluation.
Given this, there is no conceivable way that switching to Germain’s plan changes the ability to be creative.
Now, for those outsiders or people with a brand new idea absent a track record….yeah, they may take it on the chin under the current NIH system. But they would ALSO fail to gain support under Germain’s. It isn’t like we’re inventing up some new peers to do the reviewing here. No matter if it were McKnight’s panel of NAS members or Germain’s ideas of the deserving elite or traditional NIH-style panels judging the “track record”….there is no way that we can assume that genius PI behind every PCR or gene knockout technology or whatever Nobel-worthy breakthrough will be immediately recognized as awesome and funded.
JIF notes
June 24, 2015
More on NPP’s pursuit of BP is here.
see this for reference
Additional Reading:
The 2012 JIFs are out
Subdiscipline categories and JIF
Suggesting Reviewers
June 24, 2015
Who do you select when listing potential reviewers for your manuscripts?
I go for suggestions that I think will be favorably inclined toward acceptance. This may be primarily because they work on similar stuff (otherwise they aren’t going to be engaged at all) but also because I think* they are favorable towards my laboratory.
Of course.
(I have also taken to making sure I suggest at least 50% women but that is a different matter.)
I wouldn’t suggest anyone that violates the clearest statement of automatic COI that pertains to me, i.e. the NIH grant review 3-year window of collaboration.
Where do you get your standards?
___
*I could always be wrong of course
Story boarding
June 23, 2015
When you “storyboard” the way a figure or figures for a scientific manuscript should look, or need to look, to make your point, you are on a very slippery slope.
It sets up a situation where you need the data to come out a particular way to fit the story you want to tell.
This leads to all kinds of bad shenanigans. From outright fakery to re-running experiments until you get it to look the way you want.
Story boarding is for telling fictional stories.
Science is for telling non-fiction stories.
These are created after the fact. After the data are collected. With no need for storyboarding the narrative in advance.
Placeholder figures
June 23, 2015
Honest scientists do not use “placeholder” images when creating manuscript figures. Period.
See this nonsense from Cell.
Thought of the Day
June 22, 2015
I cannot tell you how comforting it is to know that no matter the depths and pedantry of my grant geekery, there is always a certain person to be found digging away furiously below me.
(in)visible at scientific meetings
June 22, 2015
Some scientists prefer to occupy scientific meeting space as the proverbial fly on the wall.
Rarely, if ever, comment at the microphone. They are not to be found gesticulating wildly to a small group of peers around the coffee table.
Others loom large. Constantly at the microphone for comment. Glad handing their buddies in every room before and after the session. Buttonholing POs at the slightest opportunity.
Someone just pointed this out to me, so I’ve been thinking about it.
Obviously nobody wants to end up being seen as a narcissistic blowhard who can’t shut up and never has anything useful to say.
But it is good to be known in your field*. And meeting visibility is part of that.
__
*Cause and effect may not be simple here, I will acknowledge.
Gender smog in grant review
June 19, 2015
I noticed something really weird and totally unnecessary.
When you are asked to review grants for the NIH you are frequently sent a Word document review template that has the Five Criteria nicely outlined and a box for you to start writing your bullet points. At the header to each section it sometimes includes some of the wording about how you are supposed to approach each criterion.
A recent template I received says under Investigator that one is to describe how the
..investigator’s experience and qualifications make him particularly well-suited for his roles in the project?
Grrr.
DrugMonkey 