Not just a list, but a story

May 23, 2014

A guest post from @iGrrrl

Like winter, changes to the biosketch are coming

Dr. Rockey spoke about the changes to the biographical sketch at the NORDP meeting this week, and I think I can at least offer a bit more depth about the thinking behind this, both from her comments and from what I’ve seen over the last few years. Certainly my knee-jerk negative opinion about this change has evolved upon reflection and listening to her presentation. This may not be as bad as it sounds. Maybe.

In her talk, of which this question of biosketches was one very small part, her short-hand way of referring to the reasoning behind this was to “reduce the reliance on publishing in one-word-named journals” as a shorthand to judging the quality of an investigator’s productivity. When the biosketch was changed in 2010, shortening the publication list to 15 seemed to me to be designed to reduce a senior investigator’s advantage of sheer numbers of publications. The rise of metrics and h-factor means that the impact factor of the journal in which the work was published now substitutes, in many a reviewer’s mind, as the quick heuristic for assessing the Investigator criterion.

The move to the shorter publication list was also borrowed from NSF’s limit of 10 products for the biosketch. This sounds good on paper, but didn’t account for the differences in culture. Researchers in NSF-type fields are just as conscious of h-index, but you don’t find the same reliance on “glamour magazines” that cut across all NSF research. The result seems to be that many young investigators in biomedicine feel they have to wait to publish until they have a story worthy of C/N/S. I hear sometimes about young researchers failing to make tenure in part because they did not publish enough, not because they didn’t have data, but because they were trying for the high-level journal, didn’t simply get it out in field- or sub-field-specific journals.

And work that appears in those so-called lower-tier journals shouldn’t be dismissed, but it often is effectively ignored when a reviewer’s eyes are looking for the titles of the high-impact journals. If a young faculty member’s list maxes to 15 and they are all solid papers in reasonable journals, that’s usually fine. But sometimes they have fewer than 15, so the reviewer relies more on the impact factor of the journals in which the work appears, and that in turn leads to reliance on C/N/S (JAMA, NEJM, etc). But for the applicant, sometimes the work reflected in the papers is based on a study that simply takes a long time to run, so that one paper in that year might represent a great deal of effort and time with results highly relevant within the context of the subfield. Or a series of papers may have methods published in one journal, the study in two more, and none of them are top-tier, but the entire story is important. This new narrative gives the opportunity to give that context.

This appears to be the point of the change to the biosketch: the impact factor of the journal(s) in which the work appeared may not reflect the impact of the results. Some applicants were including a sentence after every paper on the biosketch to try to give the context and impact–the contribution to the field–but in my experience, reviewers did not like and did not read these sentences. Yet, when reviewers come from a diversity of backgrounds, they may not be able to appreciate the impact of a result on the sub-field. Many of these concerns have been vociferously expressed to Dr. Rockey through various social media, primarily comments here at Our Host’s blot, but also on the RockTalking blog.

The idea behind this new approach to discussing an applicant’s contributions has some reasonable foundations, but I don’t expect it will work. In the short term, applicants will likely struggle to assemble a response to this new requirement. I can’t imagine reviewers will enjoy reading the resulting narratives. It may be that a common rubric approach to writing these sections as a clear story will make them uniform enough for reviewers to quickly judge, but I fully expect they will still be looking for Cell, Nature, and Science.

15 Responses to “Not just a list, but a story”

  1. @Mtomasson Says:

    Thanks for sharing the rationale about the CV proposal. Well-intentioned, to be sure, but agree strongly that a, “tell me about yourself,” paragraph is a waste of time. What real data does this provide? There’s the grant itself where one talks about preliminary data. And, it seems like we are doing everything we can to avoid the horrors of actually reading a person’s published work. The reviewer may not be in the same subfield, but I would hope they are in the same field and might consider actually reading a paper for 10 minutes. Humble suggestion for change: “If a reviewer were to read and evaluate one or two of your published papers, which ones should she read?”


  2. CD0 Says:

    Total waste of time, aimed to favor old timers and hide lack of recent productivity by holy cows.

    If I cannot determine the impact of an investigator in my field from his/her selected publications and my knowledge of my area of research, I should not be a reviewer.

    Whatever an investigator says about his/her own importance must be checked in the light of recent publications. What somebody did 30 years ago will not predict the success of a new grant better than what was published in recent years.

    As a reviewer, these changes will force me to look for recent publications in Medline and determine by myself what the likelihood of making an impact this investigator has.

    Waste of time for both reviewers and applicants.


  3. I can’t imagine reviewers will enjoy reading the resulting narratives.

    What makes you think reviewers will read them at all? As best that I can tell, reviewers never read the existing biosketch narrative, and there is no reason to think they will read this modified version.


  4. drugmonkey Says:

    I look forward to reading the paragraph long rants about how the PI submitted his or her paper to Science many times over three years while some reviewer demanded more and more experimentation. Then, of course, the paper was rejected and a nearly identical paper from some BSD lab mysteriously was accepted under a short timeline.

    I also look forward to even more grandiose claims than appear in the papers themselves about how mouse cancer has finally been cured.

    Yeah….this is going to be an awesome change to the Biosketch.


  5. Pinko Punko Says:

    Reading the current one paragraph is fine, and I absolutely don’t mind it. I think it is a place to put certain important information that goes to possible productivity issues. The long narrative- I don’t want to write it, and I don’t want to read it. It doesn’t really serve a purpose.


  6. Eli Rabett Says:

    A paragraph blurb is standard in many fields, so this is no huge change. As to the number of papers, whom amongst the panels is so lame that they can’t and don’t run citation searches on the grants they are looking at?


  7. another lurker Says:

    For my past two R01 submissions as a relatively new PI, I’ve made my personal statement 3/4 of a page on my biosketch. I describe training, including time spent in labs that aren’t readily apparent from my publications, my past accomplishments that are relevant to the proposal, and then i reiterate what my independent lab has achieved in terms of publications, preliminary data, and accolades. I can say that I do think the reviewers have read this extended blurb and that it helped (or at least didn’t hurt) because I’ve gotten mostly 1’s for my investigator scores.


  8. another lurker Says:

    Sorry, I didn’t really say what my point was — you can already write whatever you want in the personal statement, so what’s the difference?


  9. research admin Says:

    As a pre-award research admin, I am apoplectic by this news. Increasing the page limit to 5 seems like a nightmare. ESIs often feel compelled to fill any extra space–often with a 16th (and more) extra publications.

    This is an adding burden of work and time for PIs and reviewers that few can afford. Why, NIH and Sally Rockey, must you make things harder for the ESIs, the people with the most to lose???


  10. qaz Says:

    The worst of this is that it puts another burden on one’s ability to “sell oneself”. No longer can you do good science and assume that people will respect your science. No longer can you let a paper stand on its own (whether it be by journal or by assuming that people in your field have read it), now you have to explain it and its importance. This is going to highly favor people from a certain academic culture that know how to arrogantly explain how their work is the most important thing ever and has completely changed how everyone does everything, all without sounding arrogant. This is actually a very difficult line to walk.

    Some cultures brag Klingon-style by laying it all out on the table and railing about their deeds in song and story. Other cultures find that kind of a braggart insufferable and actually will think less of someone who does that. Other cultures depend on a self-depreciation as a means of putting oneself forward. All of these different cultures are highly internally successful. Now we are going to depend on a cultural match between applicant and reviewer.

    I think this is yet another misapplied fix – people are too concerned with investigator so they try to let investigators explain themselves? I’ve served on multiple study sections for over the last decade and I have never found an investigator for whom I needed an explanation. Either I know the person (or the mentor when I was on an NRSA study section) or I can track down their papers and history on this internet thing I have access to when I am reviewing their proposal.

    God, of all the problems with NIH review – this is what they focus on?


  11. AcademicLurker Says:

    At least in my experience, the investigator category rarely has a big impact on the overall score unless there’s some obvious red flag; e.g., a glaring lack of productivity, or someone who’s been a crystallographer their whole career suddenly proposing to work with c. elegans and there’s no experienced <i.c. elegans collaborator on the proposal.

    This change seems like just tinkering at the margins.


  12. Dave Says:

    I think one place where this might be useful is in training award apps, where ‘Investigator’ is probably the most important category, and where career timelines and gaps (in publications, mainly) need to be factored in. But still, not sure a one page rant is necessary.


  13. Ageing PI Says:

    I agree with the majority. As a regular NIH reviewer, I am happy looking at papers published as a gauge of PI ability/impact etc, and the brief personal statement gives me insight into the person and their career. I don’t need the PI to tell me how great they are and how important their work has been. I think I can work that out for myself.
    PS. Forward your comments to the “Rock Talk” blog and lets hope we can save ourselves all those extra pages on the next submission!


  14. E-rock Says:

    The information at NOT-OD-14-091 indicates that “up to five” contributions to science be written about (I will not say “listed” because it’s not a list), and each should not take up more than one half page, including citations and figures. The sample biosketch is an associate level professor withe three contributions described without figures and 3-5 cites each. Each a short para. I imagine that insecure noobs (like me) will be tempted to list five mid-to-lowish level contributions, create figures for each and the cites be clustered within 1-2 yr; while experienced investigators will list fewer, but broader & more sweeping contributions with cites that may span a decade. I am not sure how I feel about this. Just another thing for insecure noobs to be anxious about. But I don’t think it does anything to change how established profs are evaluated and gives them slightly more work to do. It may further pigeon hole people and make it more difficult for someone to transition from one field to another that may be a change in topic with technical similarities or to take a new perspective on a topic (lessons learned from one field applied to a similar problem in a different field) ….I don’t see that being good for Science in general, but it already happens. I also don’t see what problem this is actually addressing, were reviewers complaining that bio sketches didn’t give them enough information?


  15. E-rock Says:

    Nerd tangent — The example MS Word doc has a clickable link to a pubmed My Bibliography. Does this imply that when a grant package is put together, we can create clickable links now? I use LaTeX to write my grants, which embeds clickable links to References, Figures, or even other sections when referred to. This functionality is lost when put together in the big 424 form. Would be nice to be able to create better documents.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: