Good and bad study section sleuthing
September 21, 2011
On the one hand I’m delighted. Someone at writedit’s complains that s/he got a grant reviewed in a section that doesn’t get many grants funded at a given IC. I’m happy because it shows that this applicant is thinking strategically about the appropriate study section.
I have concerns, though. The rest of the comment seems to be blaming the IC for not being interested in the topic focus of the study section.
Hold on.
Without knowledge of the number of apps with assignment to the IC that are being reviewed in a given section, we know little. Maybe there were only five apps and three of them got funded. Maybe that other study section passed along six funded apps….but is nearly captive to the IC and reviewed 70 applications assigned to them. Better grant numbers but worse *odds* for the applicant.
I just looked at new grants for two certain ICs that arrived there through three roughly similar study sections. Two of the sections had reviewed the same number of recently funded grants for one IC1, the third was 0. Considering the other IC2, the latter reviewed about the same number of funded apps as the other two sections had funded at IC1. One of those batted zero and one sent perhaps a sixth as many to this second IC2.
So. We know we have two relatively captive sections that hand out fundable scores to IC1 and IC2 respectively. And we have a third section which reviews for both and hands out fundable scores for both.
But this is slim evidence….because of the base rate. Now I happen to know that the assignment of apps to two of the sections is also highly IC dependent….but not exclusively so. The remaining section gets mixed application assignment to IC1, IC2 and even an IC3 (substantial) and ICs4,5,6 (a handful each). (This is a very general and longitudinal/historical knowledge, btw.)
So if the mixed-assignment and the nearly-captive sections are getting the same number of apps funded at IC1…it is the *mixed* section that looks like the better bet to me. Because I assume they have fewer IC1 apps on their docket.
Let’s look at this another, bigoted way. Suppose one IC was legit, perhaps NIMH, and one was NCCAM. Would you rather your NIMH app was up against 89 other NIMH apps or up against 44 NIMH apps and 45 NCCAM apps?
OTOH what if yours was the NCCAM app? Would you rather be in a section that was practically guaranteed to hand out fundable scores to *some* applications for that IC? Or in one that could, in theory, blank that IC entirely if the apps were all worse than the top ones for a different IC?
To get back to the original comment, the point here is that you need a lot more information before you conclude a given study section is a deadend for your favorite IC. Also to realize that it may not reflect IC disinterest in the topic domain of the section that you favor.
DrugMonkey 
September 21, 2011 at 10:02 am
Do reviewers on study sections really give a shit about the IC to which the application was assigned? Aren’t all the applications thrown into the same pile? Would you favor one roughly equivalent application over another because you feel that too many applications scored well for a particular IC? I thought that a grant app is assigned a study section because of relevant expertise, not because of where it might get funded if it gets that far at all.
LikeLike
September 21, 2011 at 10:49 am
Umm, yeah. First, they are indeed supposed to care about how it fits with the mission and goals of the IC to which it has been assigned. So there’s that. Then there’s the problem of near-captive study sections being populated mostly by expertise relevant to ICx and them expressing bias about any nonICx grants that happen to stray into their study section. But that is not my point.
The NIMH/NCCAM snark was based on presuming the NCCAM grants are going to suck compared with the NIMH ones on most scientific-merit considerations. But I was trying to get you to think about more subtle versions of this…do you want to compete within-IC for an essentially field-fixed pool of fundable grant scores? or do you want to take the opportunity to try and score some “extra” high-percentile scores for your IC-based field. Or do you simply think you compete even more effectively against those clowns applying under that other IC’s mission? This is applicant based, not reviewer based, strategerizin’
LikeLike
September 21, 2011 at 11:32 am
Sure, I get that. I asked for dual assignment on my current application (only because two POs from two different institutes had interest) and ended up with assignment to three different institutes and to a study section that I think has little or no relevant expertise whatsoever. I am from the Island of Misfit Toys. I at least take comfort in the fact that the proposal is written in such a way that a fourth grader should be able to read and understand.
LikeLike
September 21, 2011 at 11:38 am
to a study section that I think has little or no relevant expertise whatsoever
did you ask the SRO for an explanation? I usually find they have a rationale for a move or apparent mis-assignment whenever I have complained. also, any sign that they brought in ad hoc reviewers just for you?
LikeLike
September 21, 2011 at 12:11 pm
DM, thanks for all the responses. Yes, I did ask for an explanation. It was along the lines of they had reviewed before in a similar research area. However, according to the RePorter website, no one in my research area that is funded was reviewed by them. I was asked to provide a list of suggested reviewers (which surprised me), but none of them, nor is anyone in a related field, included as an ad hoc so far as I can tell. They clearly had trouble assigning this application – it is multidisciplinary and unique. As I said, my only weapon is to write it for a general audience.
LikeLike
September 21, 2011 at 1:39 pm
I’ve always wanted to submit to NCCAM because I thought the competition would be so weak. Anyone with first hand experience?
LikeLike
September 21, 2011 at 3:04 pm
The study section is? Are you smoking that fucken government ditchweed again? Take a look at the official NIH peer review criteria and come back and let us know whether they say anything about the “mission and goals of the IC to which it has been assigned”.
And BTW, this post brings deranged NIH Reporter spelunking to a new level, and puts me to shame.
LikeLike
September 21, 2011 at 3:35 pm
ok, ok. so for the FOA, not the IC. same diff, practically speaking.
LikeLike
September 21, 2011 at 3:43 pm
I’ve always wanted to submit to NCCAM because I thought the competition would be so weak.
I’ve always been attracted by some of their *seeming* interest in natural products with under-evaluated pharmacology…but been scared off by what appears to be a dearth of actually-funded projects in what seem to me to be obvious domains.
Are you smoking that fucken government ditchweed again?
No, some lazy asshole is distracting me by asking me to proof his stupid grant application.
this post brings deranged NIH Reporter spelunking to a new level, and puts me to shame.
hahaha, yeah right SILK stalker.
However, according to the RePorter website, no one in my research area that is funded was reviewed by them.
Yikes, never a good sign.
I was asked to provide a list of suggested reviewers (which surprised me),
shouldn’t surprise you. it is common enough. Word to the wise, folks. when you call up an SRO to complain about having the right expertise you had better have 3-5 names you can rattle off. 3-5 names that fit into their usual criteria as well, i.e., not all noob assistant profs with no NIH funding plz.
my only weapon is to write it for a general audience.
You are on the right track. One way or another we all have to figure out how to sneak the highly esoteric shit we’re actually interested in researching into a plan that fits with funding bodies ideas of what is important….
LikeLike
September 21, 2011 at 5:17 pm
my only weapon is to write it for a general audience.
You are on the right track.
nOOO! You are writing for a specific audience, but one that doesn’t know the minutiae of your subfield. You should be *targeting* a study section that doesn’t know the minutiaeu of your dubfield.
LikeLike
September 22, 2011 at 3:58 am
This person is already scrutinizing the rosters, PP. I think the *subfield* general audience is assumed here.
LikeLike
September 22, 2011 at 11:28 am
Part of my point is that a potential strategy–one that I have employed to great effect–is to target study sections that are *not* filled with subfield experts. This scares people because, “OMFG! They won’t understand my science and why it is important!”
However, my experience is that it is a lot easier to explain your science to a non-subfield-expert audience sufficiently to get them to see its importance, than it is to convince the subfield cognoscenti that there aren’t holes in your experimental design, logic, whatever. The harshest reviewers are always the fuckebagges who know all the minute difficulties and weaknesses of your methodological and conceptual approaches.
LikeLike
September 22, 2011 at 3:30 pm
The harshest reviewers are the ones that know your proposal is bogus? I guess that makes sense.
LikeLike
September 22, 2011 at 6:25 pm
The closer the subject matter is to the reviewer’s wheelhouse, the easier it is to confuse grant review with manuscript review.
LikeLike
September 22, 2011 at 10:57 pm
Yes, I was also thinking of submitting some natural products pharmacology stuff to NCCAM. I never thought about it seriously enough to look at their funded projects.
I like your strategy, PP. I think it’s too easy for reviewers in your own niche to get obsessed with some minor technical point and not appreciate the Grand Vision.
LikeLike
September 23, 2011 at 3:57 pm
The harshest reviewers are the ones that know you’re on their jock.
LikeLike
September 23, 2011 at 7:43 pm
Isis pins the tail on the donkey.
LikeLike
September 24, 2011 at 2:34 am
Regarding NCCAM grants (as hn asked above), winning one has become exceptionally competitive. Think about how many investigators think that it’s easy money and that they could smoke the competition. But the primary reason NCCAM grants are so difficult to get is the exceptional breadth of studies the Center covers vs. a relatively paltry budget (in the ballpark of $125 million). NCCAM covers *every* alternative modality for *every* disease. It’s like having a study section called “biomedical science.”
My lab studies to anticancer and cancer preventive effects of pure compounds and complex extracts from natural sources. My track record with NCCAM has been 13 applications submitted over about six years, zero funded. I have, however, mentored two postgraduate trainees who received individual NCCAM training grants. My only “alternative” medicine funding came from NCI in the form of a R21 via their Office of Cancer Complementary and Alternative Medicine, a great model for how alt-med funding should be administered. My last R01 on an herbal product came from NCI but via a traditional cancer chemoprevention study section.
Bottom line: NCCAM is not easy money and the grants going there have been quite competitive.
LikeLike
September 24, 2011 at 7:28 am
Having served on study section numerous times, my experience is that this is not the case.
LikeLike
September 24, 2011 at 2:27 pm
CPP’s “data” bearing on this point are, of course, as anecdotal and subjective as everyone else’s. I strongly suspect that the extent to which this is true is field and study section-dependant.
LikeLike
September 24, 2011 at 4:27 pm
I hear that PP’s study section is like a carebears tea party.
LikeLike
September 24, 2011 at 5:01 pm
What is this “jock” of which you speak?
LikeLike
September 24, 2011 at 9:44 pm
I just bet it is, Isis. As re-imagined by Dante.
LikeLike
September 25, 2011 at 2:09 am
I have served on three different standing study sections over multiple cycles, and have never seen this kind of “on your jock” behavior.
Obviously, however, those three study sections are not covering wildly different areas of biomedical science. And obviously, this is anecdotal and subjective, as I explicitly acknowledged in my original comment on the subject, which referred to “my experience”.
LikeLike
September 28, 2011 at 6:04 am
Stupid question here. I must admit I’d never thought of doing this kind of sleuthing before, but as I’m preparing an R01 for next week, I’m intrigued. However, I don’t know how to determine which study sections fund work for which IC. On Reporter I can select my IC of interest, but none of the study sections I’m considering come up on the list. Why are only some study sections listed?
On another note, what are people’s opinions of special panel review vs standing study section review? (I’m likely to send this grant to a study section that I will be joining next year, so when it is revised and resubmitted, it couldn’t go back to the same study section and would have to go to a SEP. Good idea or bad idea?)
LikeLike
September 28, 2011 at 6:57 am
Good idea, anon2. Happens all the time.
LikeLike