Good and bad study section sleuthing

On the one hand I’m delighted. Someone at writedit’s complains that s/he got a grant reviewed in a section that doesn’t get many grants funded at a given IC. I’m happy because it shows that this applicant is thinking strategically about the appropriate study section.

I have concerns, though. The rest of the comment seems to be blaming the IC for not being interested in the topic focus of the study section.

Hold on.

Without knowledge of the number of apps with assignment to the IC that are being reviewed in a given section, we know little. Maybe there were only five apps and three of them got funded. Maybe that other study section passed along six funded apps….but is nearly captive to the IC and reviewed 70 applications assigned to them. Better grant numbers but worse *odds* for the applicant.

I just looked at new grants for two certain ICs that arrived there through three roughly similar study sections. Two of the sections had reviewed the same number of recently funded grants for one IC1, the third was 0. Considering the other IC2, the latter reviewed about the same number of funded apps as the other two sections had funded at IC1. One of those batted zero and one sent perhaps a sixth as many to this second IC2.

So. We know we have two relatively captive sections that hand out fundable scores to IC1 and IC2 respectively. And we have a third section which reviews for both and hands out fundable scores for both.

But this is slim evidence….because of the base rate. Now I happen to know that the assignment of apps to two of the sections is also highly IC dependent….but not exclusively so. The remaining section gets mixed application assignment to IC1, IC2 and even an IC3 (substantial) and ICs4,5,6 (a handful each). (This is a very general and longitudinal/historical knowledge, btw.)

So if the mixed-assignment and the nearly-captive sections are getting the same number of apps funded at IC1…it is the *mixed* section that looks like the better bet to me. Because I assume they have fewer IC1 apps on their docket.

Let’s look at this another, bigoted way. Suppose one IC was legit, perhaps NIMH, and one was NCCAM. Would you rather your NIMH app was up against 89 other NIMH apps or up against 44 NIMH apps and 45 NCCAM apps?

OTOH what if yours was the NCCAM app? Would you rather be in a section that was practically guaranteed to hand out fundable scores to *some* applications for that IC? Or in one that could, in theory, blank that IC entirely if the apps were all worse than the top ones for a different IC?

To get back to the original comment, the point here is that you need a lot more information before you conclude a given study section is a deadend for your favorite IC. Also to realize that it may not reflect IC disinterest in the topic domain of the section that you favor.

Overselling the candidate on a letter of recommendation

brooksphd is pondering a letter of recommendation

“X just applied and she listed you as a reference!”
…
But this feels nice! And scary – is there an added layer of responsibility on both sides of this equation now?

There is, and I observed that one should avoid overselling the candidate in making one’s comments. To this brooksphd replied:

that’s the issue I’m thinking about. Did I, could I, would I maybe oversell (or undersell) someone? Really, would it be bad to now ‘oversell’ someone? to really emphasize their fit because you can write a better letter. Is it common practice? Same as under selling someone is an easy, “I certainly consider this candidate above average. Hir fit in your lab is good. S/he reads the literature and makes solutions at the correct concentration accurately…”

Now, I recognize it is common practice to oversell and I seek ways to include a lot of confidence in the letters that I write for people. I put the best possible spin on my estimation of their talents and I may occasionally neglect to mention the odd deficit that I have observed.

But you have to keep it within reason.

I’ve had at least one experience in the past where I took someone into the lab at least partially on the strength of a recommendation letter…and this turned out to be an unreasonable oversell.

I will remind you that this is in full recognition of the type of excessive enthusiasm that we mentor types often think we need to include in the letter. Also with what I happen to think is a reasonable sympathy for the exigencies of life that can cause people’s work to be somewhat below the stellar, even for extended intervals of time.

This particular trainee sucked.

And it wasn’t just me, either. We’re talking all around failure to perform in the context of multiple obligations of this particular training dealio. It happens, and this is not the main point.

The main point is the original letter writer who testified to the skills of this particular individual in a scientific/laboratory context. There is no way in hell the letter could have been an accurate reflection. No way this person performed well in the past…or even performed at average. No way.

So my opinion of this letter writer is now and forever somewhere less than dirt. For certain sure I would never trust any other recommendations that this person might make.

I learned a lesson, my friends, a very powerful one.

You need to keep your recommendations within bounds. Do NOT ever give a glowing recommendation for someone if you know that they are going to turn out to perform significantly below average.

Because if you get burned, that mud comes back on you.