More on the Scarpa Solicitation

August 28, 2007

Another of my societies has circulated the request from Director Scarpa of the CSR to supply screened lists of senior scientists to serve on study sections. Interestingly, the head of this one has downplayed the “screened” part of the request. So far, there is no chatter on either list respecting the implications of this request.

You can see some of the motivation for appearing to include professional societies here.

Comments from Kathy Wilson of The American Society for Cell Biology:

Staffing Panels

  • Have at least 10% junior people on each section. Their freshness and honesty can counteract some of the conservatism and self-interest.

Um…wow. Somebody gets it…

But then there’s:

Dr. Gregory A. Petsko, American Society for Biochemistry and Molecular Biology

  • Assistant professors should not serve on study sections

and

Dr. Gail Cassell, American Society for Microbiology

  • No one beneath the associate professor level should serve, nor should those who have unsuccessfully competed in peer review. Peers should review grant applications.

same old drek about seniority. No explaining the reason why this is recommended…as usual.

Perhaps more telling in the Q and A:

Q: Particularly in this funding climate, it is important to use a lot of caution when using a merit-based system. We should continue to avoid cronyism and especially not bias against younger people.

No answer was supplied for Zerhouni, Tabak or Yamamoto. I’m picturing them on the dias looking at each other with blank looks…

There is still work to be done people. The comment period closes on the 7th…

17 Responses to “More on the Scarpa Solicitation”

  1. writedit Says:

    DM, we have to part company on this one. But shit, thanks for the deadline reminder. I’ve been helping everyone else here with their comments for this damn RFI … time to see if I have any original thoughts left to contribute.

    Like

  2. drugmonkey Says:

    Not sure what we are parting company on but if you are one of those enthusiastically behind this “the problem with review is all the assistant professors” thinking, well I’d like to see you explore this a bit. With respect to what impact you think asst prof level people are having on review, both in terms of numbers, real outcome and, most importantly, what detriment to the review is involved?

    It is fascinating to me that while many are willing to parrot this meme about “we need more senior reviewers” we very seldom (if ever) get a fully outlined critique of the problem, the source of the problem and hypothesized mechanism of solution…

    Like

  3. PhysioProf Says:

    I think part of what is going on is a belief that study section membership is skewed towards ineffectual dumbasses who don’t have anything better to do, and for whom study section service is their only opportunity to actually engage with a scientific community in an effectual manner. (I’m not saying I believe this to be the case; I’m just saying that this is what some believe.) And part of the reason for this is perceived to be the shift in attitude towards study section service from an honor to a thankless burden.

    But no one can come out and say this explicitly, and thereby insult those who have been serving. So instead they say “we need more senior reviewers”, with the hope that if they can get truly successful senior people to serve, it will shift the perception of service back towards being an honor. And it is hoped that this shift will attract more succcessful investigators–at all levels of seniority–into service.

    If you read between the lines of the various summaries of the meetings held so far, it sounds to me like the real issue is not perceived to be the junior/senior ratio, but the dumbass loser/successful investigator ratio. And the focus on shifting the former ratio is a “politically correct” proxy for the latter, and is hoped to have the effect of shifting the latter.

    Anyway, this is my theory on why there is a focus on junior versus senior. Again, I am not taking a position here on the merits of the underlying concern about dumbass loser versus successful investigator.

    Like

  4. whimple Says:

    I wonder if it’s as simple as the definintion of “dumbass” being “someone who didn’t fund my grant”. When paylines go from 20%+ to 10%, of course there are more dumbasses on study section.

    Like

  5. drugmonkey Says:

    Physioprof, I believe the quote is “Why am I being reviewed by some associate professor I’ve never heard of from some University in Kansas or somewhere that I’ve never heard of?”. 🙂 Agreed some of this is going on here. However this is not the thrust of the approach taken by official NIHdom, so we have to deal with the “text” so to speak. Me, I like things discussed above all else. I like people to be clear in what they are after and what they are actually proposing. So I like to challenge the conventional expressions and try to drill down to what people actually mean; the subtext. My hope is that more people will perform such challenges in the face of what I see as unthinking or at least coded statements. Clarity and coherence. Isn’t this what all scientists should be after? Why are they so bad at this when it comes to grant review approaches…?

    There is nothing wrong with coming out and saying, politely, that “too many dumbass losers are being funded and the right kind of science is suffering”. But here’s the thing. You better have game. You better not have circular definitions about “I publish in C/N/S therefore my science is great”. You had better not be woefully ignorant of what the “H” stands for or why “Institutes” is plural. You had better understand the difference between “hot” and “replicable” science. An appreciation for where our medical advances really come from. Etc.

    And whimple you have a great point but it goes beyond this. I’m very tired of people expressing the above sentiment who have never served an appointed term (or at least habitually ad hoc’d). I challenge them to go in there, do the job legitimately and thereafter maintain what I feel are their simplistic positions.

    Like

  6. PhysioProf Says:

    “However this is not the thrust of the approach taken by official NIHdom, so we have to deal with the “text” so to speak. Me, I like things discussed above all else. I like people to be clear in what they are after and what they are actually proposing. So I like to challenge the conventional expressions and try to drill down to what people actually mean; the subtext. My hope is that more people will perform such challenges in the face of what I see as unthinking or at least coded statements.”

    Actually, I believe that it not just disgruntled applicants, but also official NIHdom, that is using the junior/senior thing as code for dumbass/non-dumbass. This is my reading of the verbiage available at the “enhancing-peer-review” Web site, including the summaries of the meetings that have already occurred.

    Like

  7. whimple Says:

    DM: “And whimple you have a great point but it goes beyond this. I’m very tired of people expressing the above sentiment who have never served an appointed term (or at least habitually ad hoc’d). I challenge them to go in there, do the job legitimately and thereafter maintain what I feel are their simplistic positions.”

    Sure, I’d be happy to serve. I’m an assistant professor. Do I check that box on the application form?

    Talk about a circular argument: only those who are funded are qualified to determine who should be funded. No wonder the system invites cronyism. Excuse me for interpreting the call for “more senior reviewers wanted” as “people who owe me favors from our shared history wanted” and “people who want more of the same wanted”. *I* certainly don’t want more senior reviewers.

    It’s not the dumbass losers being funded that is killing science, it’s the PI’s that can’t keep track of how many post-docs they currently have, that have 80% of their personnel be “disposable”, that send down orders to the effect of “we need to spend more money this month”, that have 10% effort (or less) on 4 different R01s (or P01s) that are the problem. It’s the PIs that confuse *engineering* with *science* (see also: cancer genome project) that are the problem. It is the PI’s that are exemplars of the law of diminishing returns that are the problem.

    Like

  8. drugmonkey Says:

    whimple, indeed the whole relationship between synergies/economy of scale (which is a defensible position, IMO) and just-a-big-freakin-waste-of-$$ needs examination. There are just certain things that cannot be accomplished without big groups. and big groups are unbelievably inefficient. I know one lab that nearly exclusively publishes C/N/S- one data geek I know asserts this is the top 1-3 publishers of C/N/S papers all time. Because the PI refuses to put out less than overwhelmingly broad and effortful “stories” the amount of work that will never see the light of day is unimaginable. Postdocs go 7 years without first-author papers. Millions per paper. but, by our usual metrics, this is the very top science..

    study section: it is true that there is a rule that you must have a grant to serve, generally an R01 too, for that matter. I have the same problem with this that you do. There is some flexibility on the ad hoc’ing and if you have really specialized expertise that they neeed. but pertinent to Physioprof’s last point, there is an explicit effort to minimize the number of “assistant professors” reviewing grants. I talked about this coming up last fall (less than 10%) and then being dropped- well it wasn’t really dropped, just a holding battle being fought with poor SRAs trying to get grants reviewed as I hear it. the key is, whimple, for you to be one of the ones picked up “early”.

    how to “get on” study section, in brief. or at least, how to improve your chances.
    first, never refuse if you are asked to ad hoc unless there is a really good reason.
    second, there are demo- and geographic considerations so your chances increase if you are a woman, underrepresented minority or at an institution (or let’s face it, congressional district) that doesn’t have a lot of study section representation. yes, on that specific study section. so if you are the only one for several states with expertise in a major focus of the given study section, well, they need you.

    third, you have to be noticed, of course. One way is that you are submitting tons of apps through that study section and the SRA knows you- I swear my SRA gets every single recent successful applicant onto the panel for ad hoc within two rounds of them getting a fundable score! Another way is through peer-recommendations. Do you know someone one on that section? S/he may recommend you- even as a “replacement” when her/his service ends. Another is through program recommendations. I’ve talked before about the fact that POs start to get all proprietary about “their” investigators and why you should seek to build a hindbrain impression in the minds of not just one PO but several that you are one ‘o theirs. note that this will happen anyway without you doing a thing other than getting grants awarded but why not schmooze a bit and help it along?

    Like

  9. writedit Says:

    My parting company way back at the beginning was in reference to a blanket statement that more Asst Profs should serve on study sections – apparently because they’re young and cutting edge. They also don’t know what they don’t know, to quote a great Sec of Defense.

    The question is, what is the motivation for these young overworked tenure-hungry researchers to take serve on study sections? They should be publishing up the wazoo & getting their own applications in before they get shoved off the tenure ladder.

    My assumption is they want to serve so they’re part of the club – so they get exposure to how grants get funded and what grants get funded (& I’m using grants as shorthand for grant applications here) – so they get something impressive to put on their CV (for P&T & job-seeking purposes).

    Instead of committing grant applications to be reviewed as on-the-job training by junior investigators, why not set up an apprentice system whereby they can read some grant applications, attend study sections, see how the sausage is made. Once they’ve done this and had an award or two of their own (preferably had an award renewed, which demonstrates they know how to plan, conduct, disseminate, & extend a research program), they can take a seat at the table – no matter their rank, though one would hope they would have gotten beyond Asst Prof by then.

    Like

  10. whimple Says:

    writedit, you seem to discount altruism as a motive for service. If the only reason to be on study section is to get a personal benefit, what can you possibly offer senior scientists, other than the clubbiness of being able to help their buddies out?

    Like

  11. writedit Says:

    Not at all. See, the third and last paragraphs here for example. Reviewing grant applications should only be done as a service to the larger scientific community, never for personal gain. Should be.

    I just don’t think junior investigators have the time for such altruistic service but rather look to study section participation as a career development/advancement tool. I simply suggest an internship/apprenticeship approach so they can gain valuable real-world (versus word of mouth) insight into the grant process without stressing the system or themselves.

    Like

  12. drugmonkey Says:

    Thanks writedit because you express the usual conflated (sorry!) and IMHO insufficiently considered viewpoints. Yes, I’m straw-ifying you here for a point, no insult intended.

    The first thing to do is to keep distinct the effects of AsstProfs serving on study section on 1) the review of the applications and 2) the career of the AsstProf. These are distinct considerations.

    I favor increased participation of more-junior scientists on review first in service of the goal of diversity. This is using the well-tested solution of competing representative biases to counter the effects of any one type of monolithic bias. The type of bias that is a problem in my view is for equally meritorious applications from younger and older scientist to be treated differently- unjustifiably disfavoring *some* junior applicants and unjustifiably favoring *some* senior applicants. Do not mistake my position, IME more-senior people DO write better grants and less-senior people write worse ones, on average. I’m concerned with the excellent apps from untried investigators that get screwed and the clearly lesser apps from senior investigators that sail on in.
    The question of younger being more “cutting edge” is I think another plus but is not the main issue. More a defense against the “senior = better” attack. Not sure exactly where you are with “don’t know what they don’t know” but I will just remind you to always factor out actual grant reviewing experience when making judgments of who is “a better reviewer” because this is a frequently ignored flaw in the usual argument.

    As to what the AsstProf or anyone else gets out of being on study section? A much lesser issue because the focus should be on the review of grants. These issues are the back-and-fill arguments but still favor my approach. Which is NOT that AsstProfs should be obligated to serve in any way and indeed they should consider quite closely when it is time for them to serve and in what capacity. I argue only that they should not be systematically prevented.

    However, I do personally suggest AsstProfs get onto some study section service as soon as possible. For all the reasons you mention. You seem to think these are bad reasons, I call it realistic, head’s up career fostering behavior. There are many clubby and OldBoyz/Girlz aspects of this biz and it is stupid not to recognize this and how it can affect your career.
    “publishing up the wazoo” isn’t going to last long if you don’t have funding and lack of grant $$ is at least as big a problem as lacking that extra 1-2 pubs when it comes to tenure and other promotions. maybe not at all institutions and in all departments but this is reality for most these days.

    should “never be for personal gain”? c’mon. Life’s decisions and choices never have a single implication. The fact that one is judged a sufficient expert to pass muster with CSR as a reviewer is not compromised or altered because one is partially motivated to say “yes” for selfish reasons!
    Getting to know your peers in the study section environment cannot be replaced by mock-ups. And I’d doubt that any mock-up by the local institution or department really captures the real dynamics of study sections. Grant writing advice that is used by most people (writedit institutional provision of your job and expertise seems a bit unique to my eye)cannot hope to be as modern and relevant as the actual study section. There is just no substitute when it comes to the learning opportunity.

    To circle around again in anticipation, the above considerations often lead to the rebuttal “yeah but why should grant review suffer so that some AsstProf can improve his/her career status and grantwriting”. An old trick. Because, my friends, you have not established that grant review does indeed suffer on average! So we are right back to square one with the insinuated and yet rarely fully-fleshed or defended accusation.

    What “problem” with review is systematically attributable to “too many junior reviewers”, what is the evidence for this, what evidence is there that calls for more “experienced, senior reviewers” are going to change matters and what evidence that this won’t entail fully balancing if not overbalancing detrimental impacts on review?

    Like

  13. writedit Says:

    Whoa – I’ve got way too much on my desk to consider all this, but on the bike commute in, I realized I forgot to add that I do not think merely being “senior as justification for service is any more appropriate than having a quota of slots set aside for Asst Profs. I know plenty of assoc & full profs who should not be reviewing grants, either because they were never funded or last funded in the 80s or 90s (or maybe have a perpetually renewed R01 limping along) and want things to go back to the good old days. Seniority isn’t the issue – the ability to recognize proposed science that can be feasibly & practically implemented and achieve meaningful results that advance the field is.

    Like

  14. drugmonkey Says:

    “either because they were never funded or last funded in the 80s or 90s (or maybe have a perpetually renewed R01 limping along) and want things to go back to the good old days.”

    If I’m being consistent I have to argue the diversity angle in favor of assoc and full profs as well. That’s the general principle. Specifically, however, I would assert the value of the “assoc prof I’ve never heard of from some university I’ve never heard of” in the diversity equation as well. We shouldn’t just have soft-money, high pub-rate, C/N/S publishin’ people reviewing either. We need the diversity. In my area, perhaps a unique feature I don’t know, we need everything from oldskoole Exp Psych type college professor doing cheap button-pushing experiments in sophomores to gene-array mol biology jockeys. Some of the former may have had spotty NIH funding histories because their research is cheap, comes in bursts around primary teaching/service duties, etc.

    “Seniority isn’t the issue – the ability to recognize proposed science that can be feasibly & practically implemented and achieve meaningful results that advance the field is”

    Indeed. Than why do so many of the calls for improvement of grant review focus on the seniority issue? PhysioProf asserted elsewhere that this is just codewording of the real agenda. I.e., to get “hotter” scientists in and push lower-profile people out. I am not so sure. And I think the codewording, if it is such, takes on a life of its own. This is why I get up in the grill of people who seem to be expressing the “usual” complaints about AsstProf the Reviewer.

    Like

  15. AnonPI Says:

    wah, wah, Hey how about us Associate Professors? (Aren’t you one? To be on a study section for an apparently long time?)

    We need representation too!

    Like

  16. drugmonkey Says:

    Ah, yes, Assoc but relatively recently. within the past few review rounds, let us say. Reviewing grants for about a 9 round interval.

    In any case you/we AssocProf types have good representation on study section. Not much room for complaint there…

    Like


  17. […] with the approach the President had taken, I had expected a more straight-up response as requested (as another of my Societies chose to do). This particular society has now gone one additional step forward and created a “Peer Review […]

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: