Your Grant in Review: Power analysis and the Vertebrate Animals Section
February 11, 2016
As a reminder, the NIH issued warning on upcoming Simplification of the Vertebrate Animals Section of NIH Grant Applications and Contract Proposals.
Simplification! Cool, right?
There’s a landmine here.
For years the statistical power analysis was something that I included in the General Methods at the end of my Research Strategy section. In more recent times, a growing insistence on the part of the OLAW that a proper Vertebrate Animals Section include the power analysis has influenced me to drop the power analysis from the Research Strategy. It became a word for word duplication so it seemed worth the risk to regain the page space.
The notice says:
Summary of Changes
The VAS criteria are simplified by the following changes:
-
A description of veterinary care is no longer required.
-
Justification for the number of animals has been eliminated.
-
A description of the method of euthanasia is required only if the method is not consistent with AVMA guidelines.
This means that if I continue with my current strategy, I’m going to start seeing complaints about “where is the power analysis” and “hey buddy, stop trying to evade page limits by putting it in the VAS“.
So back to the old way we must go. Leave space for your power analysis, folks.
__
If you don’t know much about doing a power analysis, this website is helpful: http://homepage.stat.uiowa.edu/~rlenth/Power/
DrugMonkey 
February 11, 2016 at 2:25 pm
Not that I’ve written a ton of VAS at this point in my career (since non-NIH grants rarely require them), but when I have in the past, it was mostly a description of number of animals used in each experiment, standards of veterinary care, and euthanasia. When you eliminate these things, what do you actually write in this section?
LikeLike
February 11, 2016 at 3:41 pm
What’s a power analysis?
LikeLike
February 11, 2016 at 4:42 pm
shrew, I know right? I can’t picture what this section would even contain if not that information….
LikeLike
February 11, 2016 at 6:52 pm
I thought we just kept going until p < 0.05……
LikeLike
February 11, 2016 at 10:39 pm
Dave …I would add = and stop right there. immediately.
LikeLike
February 12, 2016 at 12:31 am
So it goes back in but counts for the now expected scientific rigor? Sheesh- it absolutely makes sense for it to be covered in the vert animal section
LikeLike
February 12, 2016 at 10:49 pm
Unless, of course, your research doesn’t involve vertebrate animals (that aren’t humans).
LikeLike
February 13, 2016 at 2:06 am
Where in the current R01 Instructions does it indicate that a Power Analysis is required? If no such requirement exists, then a grant reviewer asking for one is out of bounds. Right?
LikeLike
February 13, 2016 at 9:54 am
“Where in the current R01 Instructions does it indicate that a Power Analysis is required? If no such requirement exists, then a grant reviewer asking for one is out of bounds. Right?”
Reviewers provide expert advice to the NIH on the feasibility/impact of a proposal, and it is within their rights to ask for a power analysis if it is warranted in their judgement. There is no way for the NIH to be able to define everything that is needed for every proposal since there is such a diversity of approaches and experimental designs. Also, if NIH doesn’t penalize reviewers for dinging R21 grants for not having preliminary data when the NIH has explicitly said that R21s don’t have to have preliminary data, would you really expect them to harass a reviewer for asking for a little thing like a power analysis for the number of animals they will need?
Aren’t most VAS pretty much boilerplate, anyway?
LikeLike
February 13, 2016 at 1:46 pm
“Reviewers provide expert advice to the NIH on the feasibility/impact of a proposal, and it is within their rights to ask for a power analysis if it is warranted in their judgement.”
If the VAS instructions fail to require a power analysis, then it is an arbitrary and in my opinion bad decision on the part of a reviewer to score down a proposal for not providing one. This is an example of how study section review can be abused for the sake of a technicality.
LikeLike
February 13, 2016 at 2:33 pm
@Shrew- The VAS also contains critically important information like the scientific justification for using nonhuman animals and for the particular species used. That is, why not non-animal alternatives? (i.e., “Provide justification that the species are appropriate for the proposed research. Explain why the research goals cannot be accomplished using an alternative model (e.g., computational, human, invertebrate, in vitro).”
Also about how any potential pain and distress are minimized. (i.e., Describe the interventions including analgesia, anesthesia, sedation, palliative care and humane endpoints to minimize discomfort, distress, pain and injury.)
LikeLike
February 13, 2016 at 2:58 pm
OK, here’s a scenario. An investigator studying neurotoxicity of snorted MDMA in non-human primate model requests funds to cover 150 animals for the experiments. Is that too many animals to understand a toxic effect within the variability range of the experiment? Can you get the same data using 20 animals, thereby reducing pain and suffering for the other 120 animals? Power analysis will give you this information, which helps you better plan your experiment and maintain high ethical standards (which is something the NIH specifically asks for). Also, any institutional IACUC committee worth their salt will want this information, so it’s in your interest to show the reviewer that you’ve thought the experiment through and are asking for the resources that you actually need to get the data that you want. So expecting this type of information (while not explicitly asked for now) is a valid way to establish what kind of rigor a project (if funded) will be given.
LikeLike
February 13, 2016 at 11:27 pm
Geo- your understanding of NIH grant review (“scored down for a technicality”) is very messed up. Grants are assessed for merit in comparison with other proposals. If other proposals have features that reviewers think adds to scientific merit, it doesn’t matter that it is not mentioned in NIH’s instructions.
LikeLike
February 14, 2016 at 8:47 pm
In the SS that I used to submit to, a power analysis for every experiment was required in the Approach section. It was human and animal research. The power analysis shows that you have preliminary data that gives a rough estimate of the effect size that should be meaningful, and the power analysis justifies the sample size. You also don’t want an over powered study, i.e., one whose p < 0.05 mainly because N is so large, but the effect size (be it Cohen's D or whatever) is so small, it biologically/medically is meaningless. For animal research, I was afraid of circumventing the page limits by leaving it solely in the VAR section.
LikeLike
February 15, 2016 at 2:11 pm
I’ve always put my power analysis in the Vertebrate Animals section as well, though it’s sometimes been just to say it’s there. If part of the study has some data-driven profiling (transcriptome, proteome, etc), then it’s absurd to even do a power analysis. I suppose one could estimate the variance for 3,000 proteins that the mass spec will identify, but it’s a pretty vague estimate. An alternate approach might be one suggested by Parker and Berman: http://www.tandfonline.com/doi/abs/10.1198/0003130031919
Essentially, ask what information the study provides. To quote, “instead of ‘for difference X, I need sample size Y,’ the approach should be “for sample size Y, I get information Z.”
Still, there are some studies where power analysis really, really should be specified and having a reminder that it should no longer be in the VAS.
On the other points that DM makes, I think I’ll still practice some CYA and state explicitly that “euthanasia will follow AVMA guidelines,” rather than leaving it up for assumption.
Then again, the reviewers usually find something else they don’t like long before getting down to power analysis, in my experience.
LikeLike
February 15, 2016 at 5:02 pm
I put my spectacular, detailed power analysis, bolded, on the first page of the VAS… nobody saw it. I got slammed in review for lack of power analysis, “no details” on animal numbers, insufficient description of animal care. Everything was so carefully laid out, but I learned the cardinal rule that if it’s not where the reviewer expects it, they’re not going to see it.
Still, I’d say it’s not about *starting* to see complaints… I think this is SS specific, or perhaps is an easier critique/assumption to jump on with my ESI status.
LikeLike
February 16, 2016 at 9:17 pm
I used to get mad props for my power analyses. My department had statisticians on staff who we were required to consult in drafting our NIH grants. (and they got a chunk of the change and were expected to contribute in ms prep).
I put an overview of the results of the power analysis as a sub-sub-section at the end of the description of each experiment, and saying what it will mean for The Scienze in either direction (positive or negative results). The details were in the VAR and HS sections.
I think it is SS specific.
I never got an R01. But the positive feedback was nice. I suppose. i like peanuts.
LikeLike