I have blogged, now and again, about the ~four-tiered nature of the selection process for NIH grant awards. These consist of initial peer-review, Program Officer expression of programmatic interests, the National Advisory Council (of peer scientists) for each Institute or Center of the NIH and the Director of each IC. We tend to clump the latter three into “Program”, since they really all do express the interests of a particular IC whereas the initial peer review process focuses so strongly on the quality of the science and the strength of the overall proposal.
Unlike the apparent position of many fellow scientists, I have no problem with this multi-tiered selection process nor, more pointedly, do I have a problem when Program interests override or overturn the strict priority order / priority score that comes out of initial peer review.
I have even mentioned now and again that I think a little more transparency from Program on how they select applications outside of the review order would go a long way to damping down the whinging.
A bit of Program Officer transparency has emerged, however, that gives me pause.
September 1, 2010
Gerty-Z has a post up musing on the tiredest of StockCritiques™…”The proposal is overambitious…”.
The overall conclusion of the post, and the ensuing comments, was basically that this is totally meaningless and a grant applicant should ignore it. As Comrade PhysioProf put it:
This is pointless. The “too ambitious”/”not ambitious enough” shitte is a red herring.
I agree that this can be a meaningless, throwaway for a reviewer to put in the grant critique. But this doesn’t exactly mean that it is totally meaningless and can be ignored with impunity.
July 9, 2010
Some Twitt was asking about the importance of a Biosketch to the reviewer of a grant proposal.
Do referent letters, academic record, biosketch etc influence the application-or do you focus on the proposed research plan?
The NIH Biosketch sample Word doc file will give you an overview of the necessary components for their applications. Other funding agencies may vary in terms of what is listed so I don’t want to focus exclusively on the specific rules for NIH. Still, it is my major frame of reference.
The current NIH biosketch format leads off with a Personal Statement. This is new within the past year and nobody knows exactly how to approach this. My suggestion is that you view this as the place to write a reviewer’s bullet points on the “Investigator” review criterion for her.
As a reviewer that makes my job a bit easier but I’m not really looking at this very hard. Perhaps because it is so new.
Instead, my eye is drawn to the section that lists your employment / training stops, etc. My response to the original question is that this is highly important. I go to this either first or second (after the Specific Aims page)-not counting the title and abstract. My goal is to try to get a feel for who you are as an investigator. What your background is, what your training is…in short who you are as a scientist. (Reader whimple’s head is exploding right now.)
Why? Because it is only fair. If there is a name on the PI slot that I recognize, I already have all this information in my head. I automatically start making my adjustments, particularly when it comes to younger and less well-established investigators, in how I read the plan. If I do not recognize the name, I should try to get myself up to speed on who she is. It is, after an an explicit review criterion in the NIH system of funding.
The next section I glance at is the history of funding. If it is an Early Stage Investigator I am looking for evidence of having non-NIH research awards. The goal here is to build an argument if someone starts off on the “untried newbie” StockCritique. It helps the favorably inclined reviewer (if that is what I end up being) to have some evidence of other research projects, even if small.
If an established investigator is the PI, I glance at it but not because I think I need evidence of overwhelming support or anything. Just to orient.
Then I look at the pubs. Why come to these last? Because the frequency of pubs, frequency of first authorships, number of two-author versus multi-author and the level of journal depends to large degree on the subfield. One should calibrate ones assessment of “a productive scientist” to the demands and traditions of the subfield to the extent one can. Also to the career tenure and even the type of employment. A PI at a primarily teaching University should not be held to the same standard as someone in a research-exclusive job category.
June 22, 2010
cross posting from DrugMonkey at Scienceblogs:
I have occasionally mentioned that I really like the way that Nature Publishing Group (NPG) have promoted the online discussion of scientific research articles. After all, the publication of an article is merely the starting point and the authors’ interpretations of their data are only part of a larger set. Science proceeds best when we collaborate with our data, our ideas, our interpretations and our conclusions. Internet technologies can assist with this process. Indeed, these technologies already are assisting and have been doing so for some time. How many times in the last month have you used email to discuss a figure or a paper with a colleague? A ubiquitous phenomenon, is it not? Yeah, well when I started graduate school there was no email*.
I have also, I confess, waxed slightly critical of the execution of online paper discussion. Although I mostly bash NPG because they leave so much tasty chum lying in the water, I am generally critical; PLoS hasn’t really managed to do much better than the NPG titles when it comes to consistent online discussion.
Science blogs are slightly better at generating robust discussion of an article which in some cases feels a little more like journal club. This latter is a touchstone target for this behavior, IMNSHO. Science blogs suffer, however, from a lack of focus and a lack of comprehensive coverage. Researchblogging.org is a focal portal to select the journal article discussions out from the cacophony of a typical blog but again, it tends to suffer from coverage issues. The audience is presumed to be a general audience by most science bloggers and therefore they tend to select topics of general interest.
This brings me to a new internet creation: The Third Reviewer
The first thing you will notice is the list of journals which publish scientific articles in the neurosciences in the tabs at the top. The site grabs a Table of Contents feed and lists each article as a commentable link/entry. The comprehensive coverage problem is solved.
The site allows anonymous commenting. This is huge. It solves what I think is the major problem with the approach of publishing houses to this topic. Like it or not, people are less likely to openly comment on papers in a way that could come back to nail them. Yes, even if they are totally and completely polite, their criticism is on the up and up and 80% of the field agrees with it.
The snooty nosed types allege that anonymous commenting will make such an effort descend into meaningless drivel, ad hominem attacks and nastiness. Those of us who actually discuss papers in online venues that permit anonymous commenting allege that such risks are vastly overblown and that a light hand of moderation, plus social tone-setting, takes care of any problems that might arise.
The Third Reviewer will test these competing hypotheses. And you know I’m excited about that!
*yes, it had been developed but it was not in widespread academic use at that point.
N.b. Tragically, the owners of the movie Downfall have gone after many of the YouTube mashups, including the one from which “The Third Reviewer” derives. Has anyone seen it pop up on another host?
The font geeks are still battling it out and a recent comment on that thread got me thinking about readability. Even with the considerable limitations of Microsoft Word and my own skills with it, I’ve been able to insert figures more or less where I want them in my grant applications for years. Over a decade.
Most of the grants that I review seem to be able to manage that as well. To place and format the figures within the document text so as to, presumably, ease the job of the reviewer in apprehending the points being made. The point is to facilitate easy reference to the illustrative figure at the appropriate place in the text.
Yet manuscript review is still stuck in the dark ages. Most journal submission procedures I am familiar with still require the figures to be separate documents from the text. The figures are then appended to the back of the file when the online submission engine creates the final pdf.
Why? Why do we do this? Why not allow the authors to format the manuscript in a pdf with the figures inserted as the authors feel best? If necessary high-resolution figures could be required to be appended and the publisher could even require a parallel figure-free copy of the manuscript text for their own typesetting purposes.
April 6, 2010
A recent comment revisits a perennial issue for those new to the NIH grant game. It is initially not clear to all grant writers that you do not need to pitch your grant to an audience of all biomedicine or even to your subfield at large. You need to pitch it to a set of about 15-30 people, most of whom you know specifically because they are serving 4 (*or 6!) years terms of service on the panel. The rest you can easily phenotype by reviewing the types of individuals who have served recently in an ad hoc capacity on the panel in question. This post originally appeared July 30, 2008.
A comment from drieken on a previous post asks:
Can anyone provide some context (eg, what’s a study section that’s not in the library?) for us not-yet-researchers?
This was echoed by a recent comment over at Evil Monkey’s pad.
is there an online resource that explains the entire grant review process (NIH, NSF, whatever)?
There was also an email I received some time ago asking for an overview of the NIH system (sorry for the delay on that!).
Let’s start with the NIH study section and how you should go about educating yourself with the information that you need to guide your own grant writing.
June 12, 2008
Can anyone provide some context (eg, what’s a study section that’s not in the library?) for us not-yet-researchers?
This reminds me of an email I received some time ago asking for an overview of the NIH system. Which I haven’t done yet. (Although really people, the NIH has a very nice site on the whole process of seeking their funding for your science!)
I thought I’d start with one of my older “Your Grant in Review” posts.
As we are in the middle of study section meetings for NIH grants submitted for the June-July dates and heading toward yet another revised-application due date, I’m thinking about the way amended applications are reviewed. The amount of information available to a given reviewer on the previous history of a particular amended application is variable, leading to much dissatisfaction on the part of the applicants. The system could stand to be improved.
June 9, 2008
“As we contemplated possible changes, we were guided by several fundamental principles. First, while improving the system, do no harm. That is, ensure that any changes to the peer review system bring significant value and outweigh costs,” said Zerhouni. “Second, continue to maximize the freedom of scientists to pursue high-risk, high-impact research. Moreover, we want to cultivate a sense that we continuously re-evaluate the peer review system to ensure that it is the best that it can be.”
June 2, 2008
A now somewhat older post of drdrA’s over at Blue Lab Coats covers a recent manuscript rejection received by the laboratory. In discussing the reviewer criticisms of the manuscript, the post alludes in several places to reviewers asking for a substantial amount of additional work to be conducted. I picked up on this in a brief snark, however the critical issue was better expressed by commenter BugDoc:
I’m really concerned about what appears to be a growing trend for reviewers to ask for years worth of revisions, which could often be an additional paper. We will sometimes pull out the old standby “…beyond the scope of this paper”, but I’m curious to know if there are other rebuttal strategies with which to deflect reviews aimed at having you compress the work of an entire career into one paper.
I concur with the first sentiment, although I’d probably substitute “really, really, really annoyed” for “really concerned” if I were in a venue in which I was inhibited from expressing myself in physioproffian terms.
citations to individual articles and reviews in Nature Neuroscience (February-December, 2005) with download statistics from our website. Downloads represented the total PDF page views for any particular manuscript within the first 90 days of being posted online (including Advanced Online Publication (AOP) time).
Interesting. I’ve been pondering the potential value of article download stats for some time now so I’m intrigued by any investigation into such metrics. Perhaps this will be the start of a trend. (I will warn you in advance, however, not to expect an actual study as such out of this narrowly constrained slice of data.)
I’m mired in an effort to respond to a recent post of drdrA’s over at Blue Lab Coats on the deceptively simple issue of a manuscript rejection. This post is apparently a rich vein of blog fodder because PhysioProf already responded to the issue of pleading one’s case with the editor and I am trying to follow up on BugDoc’s request to expand on a comment I posted at Blue Lab Coats. That effort is bogging down so I thought I’d take care of one little nugget.
The part of drdrA’s post which made the most steam come out of my ears was the following point advanced by one of the paper reviewers:
“poorly performed gels and western blots which need to be improved.”
May 15, 2008
Its ok, these things happen and its just a paper. I’m not really upset about it that much and will turn it over somewhere else.
Hey! Not so fast with the resignation! One thing I have learned over the years is to never take a paper “rejection” as a rejection until an editor tells you personally–not using automated boilerplate language–that she absolutely refuses to reconsider the paper.
The NIH grant applications which will be reviewed Jun/Jul are going out to reviewers right about now. Poking through my pile of assignments I find that I have three R01 applications at the A2 stage (the second and “final” amendment of a brand new proposal). Looking over the list of application numbers for the entire panel this round, I see that we have about 15% of our applications on the A2 revision.
Oi. What a waste of everyone’s time. I anticipate many reviewers will be incorporating the usual smackdown-of-Program language. “This more than adequately revised application….”
April 23, 2008
Bora linked to a recent article at the Scientific American Web site concerning “Web 2.0″ and its relationship to the conduct of science. The basic concept is that “THE WEB TOTALLY CHANGES EVERYTHING!11!!!!!1!ELEVENTY!11!!!!!1!” and scientists are and/or should be going to completely alter their way of communicating their findings to one another. One aspect of this is the notion of “Open Access Lab Notebooks”, pursuant to which scientists will keep their lab notebooks on publicly accessible Web sites where other scientists can view them and update the information they contain on a daily basis. In other words, scientists will essentially be continuously live-blogging their experimental activities in the lab. This is a totally fucking stupid idea.
[The motivating context for this has been removed because subsequent developments made it obvious that it was such a unique event that my post would violate the confidentiality of all concerned. In both minor and major ways. -DM]
I keep meaning to talk about “member conflict” SEP review and there is no time like the present.