GrantRant VII
January 10, 2013
Everyone is going to hate you, pretty much.
Think about it. You have 7-10 grants assigned in your pile on a typical study section these days. Odds are good that at best one or two of these is going to be good enough to be in the hunt for funding. The rest of the panel is in the same boat, so it really doesn’t matter that the applicants don’t know precisely which of you* on the panel reviewed his or her proposal.
80-90 % of the applicants are going to be mad at you.
Since you have been selected for expertise in the relevant field…these are people who you know. You know their work and you probably like and cite it. They know you. They know your work.
And for at least a while after they see their disappointing score, and for another while after the pink sheets are posted, they cannot help but hate you a little.
Maybe even a lot.
Joyous.
__
*If you were triaged you do know for absolute sure that every member listed on that panel roster stood by and refused to pull your application up for discussion.
DrugMonkey 
January 10, 2013 at 12:25 pm
True, and really unpleasant to consider.
LikeLike
January 10, 2013 at 12:28 pm
The disgruntleranting about UNFAIR BIAS AND FACTUAL ERRORS OF UNETHICAL REVIEWERS KILLED MY GENIUS GRANT that is selected for in the blogges is actually pretty rare in real life, as far as I can tell. The people in my fields whose opinions of me matter are–with a few rare exceptions (and these are, themselves, without exception those whom everyone else knows to be hotheaded delusional cranks)–rational enough to understand how these things work and not to “hate” me when their grant that is reviewed by my study section doesn’t get a fundable score.
LikeLike
January 10, 2013 at 12:28 pm
The bad part is that there are enough grants in my area that I am not assigned many of the grants where people would assume that I was a reviewer. Also, when the three reviews agree, I can’t personally be ticked at the panel. I have heard from people that they regret being on study section because they feel like the field has essentially cooled to them and they attribute it directly to their section service.
Question for DM: do you feel nervous about speaking up about a grant in the negative when discussion is contentious? Do you have any worry that against all regulations, it might get back to that person that you were a possible mover on their score? This is a particularly negative situation because it undermines the confidentiality of the entire process.
LikeLike
January 10, 2013 at 12:30 pm
This is one of the pleasures of not being in the US funding system. There are enough funds for anyone wit a half decent chance of success, so you really aren’t saying no to anyone who didn’t deserve it.
This has also been discussed here before, I think: the high rejection rate leads to more resubmissions which leads to a higher rejection rate. If people had a maximum of two rejections a year they would submit their proposal to a well chosen program instead of across the board, thus automagically doubling the acceptance rate without a single extra penny in increased funding.
LikeLike
January 10, 2013 at 12:36 pm
“REVIEWERS KILLED MY GENIUS GRANT that is selected for in the blogges is actually pretty rare in real life, as far as I can tell.”
That has been my experience with the vast majority of “pink sheets” I’ve received, be them grants or papers. Almost all of them had insightful comments reasonably justifying their respective negative decision.
However, it must be said that something like 1 out 10 negative decisions I gotten over the years are relatively random and about half of those are completely bat-shit crazy. I don’t think this is that surprising either, given the restrictions the system operates under, namely time pressures, unintended biases, and yes the occasional dishonest reviewer.
LikeLike
January 10, 2013 at 12:44 pm
I have heard from people that they regret being on study section because they feel like the field has essentially cooled to them and they attribute it directly to their section service.
I have never heard anything like this from any of the many dozens of people I know who have been charter members of a number of different study sections.
LikeLike
January 10, 2013 at 12:44 pm
I agree that the “REVIEWER…” thing is very rare, but the being pissed about a slip from a 2 to a 3 is not and that is the make or break for all grants now. Either way, this stuff is no fun for anyone when hardly anything gets funded and everyone knows its impossible to distinguish meaningfully among the top 20%. There is nothing worse than reviewing a huge stack of grants and knowing at the end of the day that none of them are going to happen.
LikeLike
January 10, 2013 at 12:48 pm
This is why junior people need to sit on study sections – so they know how they work, and understand it is NOT ABOUT THEM. Even more so lately (last 2-5 years) than in the past, study sections are made up of people who know this is going on, and do try to be as fair and honest as they can. It takes more than one person to either sink a grant or float it to the top.
If after watching a study section in action you still think that “UNFAIR BIAS AND FACTUAL ERRORS OF UNETHICAL REVIEWERS KILLED MY GENIUS GRANT” then you are not playing with a full deck. Or rather a deck that is full of it.
LikeLike
January 10, 2013 at 12:56 pm
“then you are not playing with a full deck. Or rather a deck that is full of it.”
You mean you’ve never seen a fellow panelist/reviewer go bat-shit crazy for no apparent reason on a grant? I’ve can recall of the top of my head at least two instances (out of say hundreds of grants/papers reviewed) in which the criticisms seemed to come out of a personal bias instead of anything in the proposal itself.
In other words, it does happen, just not very often, at least in my experience, and lastly for certain programs a single negative opinion can sink a grant. Depends what pot of money you are applying for and how competitive the field is.
LikeLike
January 10, 2013 at 2:10 pm
CPP- the only one I trust not to be a hotheaded delusional crank is myself 🙂
Question for DM: do you feel nervous about speaking up about a grant in the negative when discussion is contentious?
It turns out that I am entirely incapable of shutting up. I’m sure y’all are shocked.
worry that against all regulations, it might get back to that person that you were a possible mover on their score?
I pretty much assume that with all people’s assertions that they “know” who it was, there is very little difference in this regard.
I have never heard anything like this from any
Yeah, me neither. And I made so many new friends and acquaintances through study section service. from meeting old guard types to cementing relationships with some people more my age…it is always a good thing.
You mean you’ve never seen a fellow panelist/reviewer go bat-shit crazy for no apparent reason on a grant?
What I can assert is that I’ve never seen a case where a reviewer going bat-shit crazy on a grant was the sole reason for it getting hammered when it would otherwise have been funded.
LikeLike
January 10, 2013 at 2:19 pm
mentors told me I should wait a while to get on study section, because I am just going to make lots of enemies, and to do it too early would be trouble.
LikeLike
January 10, 2013 at 2:20 pm
btw, jp, where you been man? we’ve missed you!
LikeLike
January 10, 2013 at 2:21 pm
because I am just going to make lots of enemies
as you know I disagree and think the benefits outweigh the potential for “enemies”.
LikeLike
January 10, 2013 at 3:00 pm
I would love to be on a study section. It would improve my grants immeasurably. I have no doubt about it.
LikeLike
January 10, 2013 at 3:08 pm
I am definitely more vocal than average member.
The people who feel the way I alluded to above are in a smaller field and I think they feel they are the token member of that subfield on the panel. I am lucky enough to be in a bigger field and there are usually 3-5 people each time for sure that are related to this field.
I try to review for both the applicant and the panel. I know that DM has talked about that the guidelines are to give guidance to the panel, but I want to read thoughtful/helpful reviews on my own grants, so that is what I try to do for the ones I get.
LikeLike
January 10, 2013 at 3:24 pm
There are certainly worse things than the Golden Rule to use for a rubric
LikeLike
January 10, 2013 at 3:55 pm
I too think the positives outweigh potential negatives. Just chiming in that more than one really good scientists and friend has said this. they were all on the same study section…..
LikeLike
January 10, 2013 at 5:20 pm
been writing grants and serving on study section 🙂 and hanging out with your buddy on FB
LikeLike
January 11, 2013 at 5:38 am
In my experience, the “haters” are delusional regardless.
The ones who come up to you at a conference, bitching about study section, are pretty much the ones who bitched about study section before I was on it, and will keep doing so long after I’m gone from study section. I’m just a convenient target for their rage.
LikeLike
January 11, 2013 at 7:13 am
pinus-
If we’re talking topic domain A, I’m starting to get a paranoid smell from those guys. An *amazingly* paranoid smell given how successful the whole bunch of them have been at staying funded….
LikeLike
January 11, 2013 at 11:29 am
“Since you have been selected for expertise in the relevant field…these are people who you know. You know their work and you probably like and cite it. They know you. They know your work.”
Is this really how it’s supposed to work? I might be screwed. I just applied for my first big grant as an assist prof and don’t recognize any of the names on my study section. I didn’t know any of the names on the study section that funded my F32 either. Both study sections were dominated by MD types or PhDs doing bench work. My work involves more computation, stats, and generally different approaches to questions that are definitely under the NIH purview but barely overlap with any of the research done by section members. Two of the criticisms I received on my F32 implied the reviewers hadn’t read anything in my field and didn’t realize what the norms are–the comments weren’t even from left field, it was more like another galaxy. What do I do about this?
LikeLike
January 11, 2013 at 2:16 pm
You need to talk to SROs in advance, scrutinize the rosters, check reporter for work like yours, etc. if you really can’t find good expertise find the closest panel and then lobby the SRO about the type if expertise that is needed.
LikeLike
January 11, 2013 at 2:18 pm
…but yeah, this is part of what slants the table against the newer applicants. Publishing helps. Attending meetings helps. …..those take time.
LikeLike
January 12, 2013 at 5:00 am
“I just applied for my first big grant as an assist prof and don’t recognize any of the names on my study section.”
If this means there is absolutely no one on the panel who is qualified to review your grant, then you are screwed. You need to research study section composition BEFORE you submit. Find the one with the most relevant expertise, then talk to your PO and SRO about it, and then indicate on your cover letter which study section you think it should go to. And then complain like hell if it’s still not assigned to the one you want.
If, instead, your statement means that there are people on the panel who are qualified to review your grant (i.e., they have the relevant expertise), but you happen not to know any of them personally, then that’s not so bad. It certainly helps to know people, but it’s not as if reviewers only give good scores to people they know (as DM keeps pointing out, because, you see, according to him the whole system is only minimally influenced by “bias” of this type).
But even DM agrees that you need to get your name out in front of these people. Going to meetings is a start, but you should also consider inviting people in your field (and slightly out of it) — the kind of people who might review your grants — to give a seminar in your department. That way you get to spend a lot of time with them, they get to learn about the exciting stuff going on in your lab, and the whole thing ends on a warm fuzzy note with you treating them to dinner. Naturally this won’t influence them AT ALL the next time they review your grant, because of the lack of the aforementioned “bias,” but it certainly can’t hurt.
(On a related note, it was recently pointed out to me that the PI of a huge center grant on my campus, who is also a department chair, invited every single member of the reviewing study section to give a talk in the year prior to submitting the renewal. Hmm.)
LikeLike
January 12, 2013 at 5:57 am
What I assert Grumbie, is that you can never know for sure who your “friends” and “enemies” are for any given grant proposal review. I do not suggest that personal warm-fuzzy bias doesn’t exist. Indeed I’ve written a few times about the very power of the personal connection. Particularly when it comes to POs but also as a part of my advice to keep hammering a given study section with proposals. They “get to know you” through the apps, it isn’t just about in-person schmoozing.
Inviting people for talks only b/c they are on study section is a step too far for my taste but it certainly happens. Again, I’m skeptical about the value.
LikeLike
January 12, 2013 at 9:14 am
It may seem a bit sleazy to invite only study section members to give talks, especially those whose work you couldn’t really care less about. But one thing to keep in the back of one’s mind, when thinking of whom to invite, is how likely someone is to review your grant.
LikeLike
January 12, 2013 at 1:13 pm
DM, It was said topic domain. not reviewing that one though.
LikeLike
January 13, 2013 at 12:43 pm
“You need to research study section composition BEFORE you submit. Find the one with the most relevant expertise, then talk to your PO and SRO about it, and then indicate on your cover letter which study section you think it should go to.”
Unfortunately, not all grants permit this level of specification. I have seen people with approaches very roughly along the lines of mine get this award, which gives me a teensy bit of hope… but I think the real issue here is that I should have contacted the PO and SRO in advance, and maybe they could have shifted the study section composition. (Is that how it might work?)
I am getting discouraged from seeing so many rosters of reviewers without backgrounds similar to mine. I don’t even know how I would “sell” my approaches and methods to potential section members–we’re clearly going to different conferences, although I’m still getting pubs in broadly respected journals. It’s hard to sell people who study the detailed interaction of X and Y in animal model Z on inferential, less reductionist approaches from other fields. There seems to be a sharp falling off of interest at a certain methodological distance.
LikeLike
January 13, 2013 at 12:56 pm
p.s. I had assumed that because the roster wasn’t posted until two months after the grant deadline, the SRO would be taking the range of application topics into consideration when inviting section members. DM, you use the word “lobby,” which implies I should’ve emphasized the kind of expertise suitable for an app like mine. Is it worth trying now? (No meetings yet.) Damn damn damn.
LikeLike
January 13, 2013 at 1:39 pm
It’s probably too late for the SRO to change the study section composition now. I’m not sure that contacting him/her would do any good at this stage. But, once you get the reviews, assuming they’re not very enthusiastic, it wouldn’t hurt to contact the SRO to mention that you don’t understand why no one on the panel had expertise in your particular approach. If you have a good PO, he/she will also be receptive to this complaint and make a point of talking to the SRO about it (my own PO, who is excellent, has done this for me).
Here’s an idea – I’m not saying it’s a good idea, but I’ll throw it out there for those with more experience to comment on. Would it make sense to call the SRO and PO now, discuss the lack of expertise issue, and if you get sympathetic noises from the SRO now, withdraw the grant and submit in the next cycle (Feb 5 deadline, or Mar 5 for resubmissions)?
“I have seen people with approaches very roughly along the lines of mine get this award …” If you know these people well, ask them which study section reviewed it, and for a copy of the roster (it’s tacked to the end of the summary statement). Seems like that would be very useful information.
LikeLike
January 14, 2013 at 4:00 am
@Grumble – Complaining about expertise is one thing that PO’s and SRO’s get lots of flack on. It’s important, but also important to tread carefully and as always, request, don’t demand.
It is also possible to add reviewers, usually phone reviewers, at a late date to change the composition of a study section. I’ve been asked last minute for this exact reason. However, phone reviewers (in general) can be off the scoring of the study section (no data here, just impression) as they are not part of the entire show.
As for recognizing names, that’s what the internet is for. Lots of times I’ve not recognized the names, but with a little research realized that the folks are either appropriate (and I’ve discovered new stuff, indexed under different key words) or claiming expertise in the area. The latter means that the PO or SRO is not going to take much effort in listening to your complaints.
Finally, remember, never ever suggest names to the PO/SRO. They are not permitted by NIH rules to solicit names from people who submit, and react very badly to those who do.
LikeLike
January 14, 2013 at 4:16 am
I have had a different experience than Potnia regarding name suggestion. When I call and whine to SROs about specific expertise I definitely have names to drop. And I have indeed seen them show up on panels subsequently. Now maybe things have tightened up lately…
Damn- yes the SRO does take the current panel load into account. But these days they are being limited in ad hocs compared to times past. And, anyway, looking at the past three rounds’ rosters (which are posted in some CSR page, google roster and the study section to find them) will give you some idea of the types of people who will be called. That’s the info you want to use on the SRO *immediately after the proposal has been assigned to a study section*, to argue your concern about expertise.
If you see funded grants like yours, RePORTER lists the study section it was reviewed in. This is an essential factor in deciding which study section to request in your cover letter.
LikeLike
January 14, 2013 at 4:21 am
Grumble-
Talk to the SRO about panel composition, not the PO. The former are not keen about the latter sticking their nose into the review side. I get a vibe not dissimilar to the business/journalist divide at a news organization.
LikeLike
January 14, 2013 at 4:34 am
Another thought for Damn-
You do realize that NIH panels are in most cases composed mainly of a pool of empaneled people serving a fixed (four to six) term of service? And that the number of people brought in as “ad hocs” for each round to cover expertise gaps relevant to the specific applications is limited?
If you perceive that you are never getting the right reviewers (yet they exist in the potential pool of already NIH-funded investigators) then you may need to research panels a little harder.
LikeLike
January 14, 2013 at 6:30 am
DM, Grumble, & potnia, thank you for the suggestions and information.
The grant I applied for isn’t a R01, and my grant has been assigned to a Special Emphasis Panel of temporary members. But there are also supposed to be several phases of review, and I’m not certain from the grant description if people outside this particular SEP will provide comments. It’s also not obvious to me whether alternative SEPs exist for this grant (although I can at least filter out the ones meeting in a different month). Assuming there aren’t major objections here, I think I will email the SRO and inquire whether a few of the other SEPs listed are options and might be better fits.
LikeLike
January 14, 2013 at 6:54 am
I’ll also mention to the SRO that I’m hoping to find a section with someone who has background/expertise in a certain area.Figuring out the structure of these SEPs has been embarrassingly hard… I checked RePORTER to see which study sections recently funded awards in my area, but I can’t find any evidence the sections still exist.
LikeLike
January 14, 2013 at 7:17 am
Ahh yes, the Special Emphasis Panel is a much trickier beast. Talking to the SRO (and the PO for the RFA, if this is what the reason is for SEP) may be your only option.
LikeLike
January 14, 2013 at 7:18 am
And obviously, this uncertainty is why you can’t live by SEP alone….
LikeLike
January 14, 2013 at 7:57 am
” there are also supposed to be several phases of review, and I’m not certain from the grant description if people outside this particular SEP will provide comments. ”
Generally, “several phases of review” just means that once the scientific review panel gives the grant a score, the program staff then decides whether to fund it and if so, at what level. Their decision is supposedly then reviewed by the institute’s advisory council (this review is usually pretty much a rubber stamp.) After that, the grants management staff performs “administrative review,” which means things like making sure you have animal or human subjects approval, making sure there is no scientific or budgetary overlap with any of your other grants, etc.
So, as you can see, the most important thing is to get a good score; everything else is just pro-forma, with the one exception being that if you have a “gray zone” score, the program staff decides whether to fund. Therefore, having the right review panel composition is very very very important, and no amount of secondary review is going to change the outcome if that panel decides it doesn’t like your grant. (In other words, it’s *extremely* rare for Council or POs to resurrect a grant with a bad score. Council essentially never does it, and POs almost never consider a grant with a score worse than the upper limit of the “gray zone” – usually around the 20th percentile, but maybe lower nowadays.)
LikeLike
January 14, 2013 at 8:14 am
Grumble-
It is possible that this person is in one of these pilots for manuscript style review that were started around the ARRA time. Initial readers who don’t meet, then a study section that is like an editor board, sort of. Only thereafter to Council…
LikeLike
January 14, 2013 at 8:18 am
Thanks again for the explanations. I can’t think of a kinder, faster way to learn this stuff as a n00b.
LikeLike
January 14, 2013 at 8:28 am
Well, ideally you’d have some local associate profs to give you a clue on this stuff….
LikeLike
January 16, 2013 at 4:58 am
SEP’s and ad-hocs are problematic. Some SEP’s exist for mechanisms that are handled by IC’s (ie K’s, F’s, T’s, and some R03’s) and not by CSR. You can often find these lists at the IC homepage. (If this acronyms have your confused try: http://grants.nih.gov/grants/acronym_list.htm). Damn – what mechanism is this application?
DM has the right idea about checking IRG compositions. Start here: http://public.csr.nih.gov/StudySections/Standing/Pages/default.aspx
There are some good cross links to “related sections” that make it easier to search. Remember that you can always ASK for an assignment in your cover letter – either to institute or to IRG.
And, while Grumble is correct about the standard two phases (Study Section, Institute) for nearly all grants, I have recently reviewed a couple of grants that had two phases of study section, both SEP. These were not R01’s but a different mechanism within NIDCR. I did a science level review (online section- NIH does chat rooms) and then there was a clinical relevance review. I’m trying to remember, but I think it was a big phase3 clinical trial. It was odd, and we were given very specific guidelines as to what to review.
LikeLike
January 18, 2013 at 8:19 am
This is why reviewers need to carefully think about their criticisms and write thoughtful helpful reviews.
Review — whether for funding agencies or journals — is not about crapping on people’s efforts or serving as a gatekeeper. It’s about helping colleagues produce the best work possible.
Who doesn’t like that?
LikeLike
January 18, 2013 at 8:50 am
It’s about helping colleagues produce the best work possible.
Nope. It’s about helping the funding agency select the most meritorious science (by their needs/desires) for them to fund.
Thinking it is to help improve your colleagues’ science is a perversion of this process
LikeLike
January 18, 2013 at 10:48 am
@Dave: wrong, wrong, wrong. DM is right, right, right. In fact, if you are on a study section and you start to make suggestions about how to improve the project your wrist will be slapped (if the SRO is any good) and told “it is not our job to improve science, only to evaluate what is proposed”.
That’s what is wrong about getting mad about reviews: if you expect help you are NOT going to get it.
LikeLike