You know the old story.

In this new story, we have the NIH’s Sex As a Biological Variable (SABV) policy. When first discussed, just about everyone who took this seriously pointed out the problem of a zero sum, limited funding system adopting a mandate which would double the animal costs. To really consider SABV properly, we said, this is going to double our sample sizes…at the very least. Probably more than double.

That is coming from the perspective of a scientist who works with units of the whole experimental animal. There are many of us.

The official NIH response was a bunch of gaslighting.

“Oh no”, went the policy mavens of the NIH, “this is not what this means at all. Simply include equal numbers of male and female animals at your regular sample size. That’s it. Oh, yeah, you have to say you will stratify your data by sex and look at it. You know, just in case there’s anything there. But nothing insists you have to double your sample size.”

Sure, said we NIH watchers/applicants. Sure it will go like that. Have you met our reviewers? They are going to first of all demand that every study is fully powered to detect any sex difference. Then, they are going to immediately start banging on about swabbing and cycling the female rats and something something about powering up for cycle as well.

NIH: “No, of course not that would never happen why we will tell them not to do that and everything will be copacetic”

Things were not copacetic. As predicted, reviewers of grants have, since even before the mandate went into effect, demonstrated they are constitutionally unable to do what NIH claimed they should be doing and in fact do what they were predicted in advance to do. Make everything HAVE to be a sex differences study and HAVE to be a study of estrous cycle. Randomly. Variable. Yes. As with everything in NIH review. And who knows, maybe this is a selective cudgel (I call it Becca’s Bludgeon) used only when they just generally dislike the proposal.

The NIH mandate let the SABV camel’s nose under the tentflap and now that camel is puuuuuuuuussssshhhhing all the way in.

A new article in eLife by Garcia-Sifuentes and Maney is part of this campaign. It is chock full of insinuations and claims trying to justify the full camel in side the tent. Oh, they know perfectly well what the NIH policy was. But they are using all of the best #allegedprofession techniques to try to avoid admitting they are fully doing an end run.

From the Abstract: This new policy has been interpreted by some as a call to compare males and females with each other.

From the Intro: Although the NIH policy does not explicitly require that males and females be compared directly
with each other, the fact that more NIH-funded researchers must now study both sexes should lead to an increase in the frequency of such comparisons (insert self-citation). For example, there should be more testing for sex-specific
responses

“should”.

although the proportion of articles that included both sexes significantly increased (see also Will et al., 2017), the proportion that treated sex as a variable did not. [Note interesting goalpost move. or at least totally undefined insinuation] This finding contrasts sharply with expectations [whose “expectations” would those be?], given not only the NIH mandate but also numerous calls over the past decade to disaggregate all preclinical data by sex [yes, the mandate was to disaggregate by sex. correct.] and to test for sex differences [bzzzt, nope. here’s another slippery and dishonest little conflation]

One potential barrier to SABV implementation is a lack of relevant resources; for example, not all researchers have received training in experimental design and data analysis that would allow them to test for sex differences using appropriate statistical approaches. [oh what horseshit. sure, maybe there is a terrible lack of experimental design training. I agree those not trained in experimental psychology seem to be a bit lacking. But this is not specific to sex differences. A group is a group is a group. so is a factor. the “lack of relevant resources” is….money. grant money.]

any less-than-rigorous test for sex differences creates risk for misinterpretation of results and dissemination of misinformation to other scientists and to the public [There you have it. The entire NIH scheme to introduce SABV is not only flawed, it is, seemingly, even worse than doing nothing!]

Although a sex difference was claimed in a majority of articles (57%), not all of these differences were supported with statistical evidence. In more than a quarter of the articles reporting a sex difference, or 24/83 articles, the sexes were never actually compared statistically. [Yep, totally consistent with the assertions from NIH about what they were after. Anything else is a significant move of the goalposts. In the direction that was anticipated and EXPLICITLY denied as being the goal/end game by the NIH. In oh so many ways.]

In these cases, the authors claimed that the sexes responded differentially to a treatment when the effect of treatment was not statistically compared across sex. … Of the studies with a factorial design, 58% reported that the sexes responded differently to one or more other factors. The language used to state these conclusions often included the phrase ‘sex difference’ but could also include ‘sex-specific effect’ or that a treatment had an effect ‘in males but not females’ or vice versa. … Neither approach tests whether the treatment had different effects in females and males. Thus, a substantial majority of articles containing claims of sex-specific effects (70%) did not present statistical evidence to support those claims

This is also, utter a-scientific horseshit.

I get this a lot from reviewers so I’m going to expand but only briefly. There is no such thing as canonical statistical interpretation techniques that are either “right” or “wrong”. Nor do statistical inference techniques alter the outcome of a study. The data are what they are. All else is shades of interpretation. At the very best you could say that different inferential statistical outcomes may mean there is stronger or weaker evidence for your interpretations of the data. at best.

But there is a broader hypocrisy here. Do you only build your knowledge within the context of one paper? Do you assemble your head space on whether something is likely or unlikely to be a valid assertion (say, “female rats self-administer more cocaine”) ONLY on papers that provide like-to-like perfectly parallel and statistically compared groups?

If you are an idiot, I suppose. Not being an idiot, I assert that most scientists build their opinions about the world of science that they inhabit on a pile of indirectly converging evidence. Taking variability in approach into account. Stratifying the strength of the evidence to their best ability. Weighting the results. Adding each new bit of evidence as they come across it.

And, in a scenario where 10 labs were conducting cocaine self-administration studies and five each tended to work on males and females independently, we would conclude some things. If we were not preening Experimental Design Spherical Cow 101 idiots. If, for example, no matter the differences in approach it appeared that in aggregate the females self-administered twice as many infusions of the cocaine.

We would consider this useful, valid information that gives us the tentative idea that perhaps there is a sex difference. We would not hold our hands over our eyes mumbling “blah blah blah I can’t hear you either” and insist that there is zero useful indication from this true fact. We would, however, as we do with literally every dataset, keep in mind the limitations of our inferences. We might even use these prior results to justify a better test of the newly developed hypothesis, to overcome some of the limitations.

That is how we build knowledge.

Not by insisting if a comparison of datasets/findings does not accord with strict ideas of experimental design rigor, it is totally invalid and meaningless.

Among the articles in which the sexes were pooled, the authors did so without testing for a sex difference almost half of the time (48%; Figure 3B). When authors did test for a sex difference before pooling, they sometimes found a significant difference yet pooled the sexes anyway; this occurred in 17% of the articles that pooled.[Yes, consistent with the NIH policy. Again with the moving the goalposts….]

Thus, the authors that complied with NIH guidelines to disaggregate data usually went beyond NIH guidelines to explicitly compare the sexes with each other. [hookay…..so where’s the problem? isn’t this a good thing?]

There is a new review by Shansky and Murphy out this month which addresses the NIH policy on considering sex as a biological variable (SABV).

Shansky, RM and Murphy, AZ. Considering sex as a biological variable will require a global shift in science culture. Nat Neurosci, 2021 Mar 1. doi: 10.1038/s41593-021-00806-8. Online ahead of print.

To get this out of the way, score me as one who is generally on board with the sentiments behind SABV and one who started trying to change my own approach to my research when this first started being discussed. I even started trying to address this in my grant proposals several cycles before it became obligatory. I have now, as it happens, published papers involving both male and female subjects and continue to do so. We currently have experiments being conducted that involve both male and female subjects and my plan is to continue to do so. Also, I have had many exchanges with Dr. Shansky over the years about these issues and have learned much from her views and suggestions. This post is going to address where I object to things in this new review,for the most part, so I thought I should make these declarations, for what they are worth.

In Box 1, the review addresses a scientist who claims that s/he will first do the work in males and then followup in females as follows:

“We started this work in males, so it makes sense to keep going in males. We will follow up with females when this project is finished.” Be honest, when is a project ever truly finished? There is always another level of ‘mechanistic insight’ one can claim to need. Playing catch-up can be daunting, but it is better to do as much work in both sexes at the same
time, rather than a streamlined follow-up study in females years after the original male work was published. This latter approach risks framing the female work as a lower-impact ‘replication study’ instead of equally valuable to scientific knowledge.

This then dovetails with a comment in Box 2 about the proper way to conduct our research going forward:

At the bare minimum, adhering to SABV means using experimental cohorts that include both males and females in every experiment, without necessarily analyzing data by sex.

I still can’t get past this. I understand that this is the place that the NIH policy on SABV has landed. I do. We should run 50/50 cohorts for every study, as Shansky and Murphy are suggesting here. I cannot for the life of me see the logic in this. I can’t. In my work, behavioral work with rats for the most part, there is so much variability that I am loathe to even run half-size pilot studies. In a lot of the work that I do, N=8 is a pretty good starting size for the minimal ability to conclude much of anything. N=4? tough, especially as a starting size of the groups.

The piece eventually gets around to the notion of how we enforce the NIH SABV policy. As I have pointed out before and as is a central component of this review, we are moving rapidly into a time when the laboratories who claim NIH support for their studies are referencing grant proposals that were submitted under SABV rules.

NOT-OD-15-102 appeared in June of 2015 and warned that SABV policy would “will take effect for applications submitted for the January 25, 2016, due date, and thereafter“. Which means grants to be reviewed in summer 2016, considered at Council in the Fall rounds and potentially funded Dec 1, 2016. This means, with the usual problems with Dec 1 funding dates, that we are finishing up year 4 of some of these initial awards.

One of the main things that Shansky and Murphy address is in the section “Moving forward-who is responsible?“.

whether they have [addressed SABV] in their actual research remains to be seen. NIH grants are nonbinding, meaning that awardees are not required to conduct the exact experiments they propose. Moreover, there is no explicit language from NIH stating that SABV adherence will be enforced once the funds are awarded. Without accountability measures in place, no one is prevented from exclusively using male subjects in research funded under SABV policies.

Right? It is a central issue if we wish to budge the needle on considering sex as a biological variable. And the primary mechanism of enforcement is, well, us. The peers who are reviewing the subsequent grant applications from investigators who have been funded in the SABV era. The authors sortof mention this: “Researchers should be held accountable by making documentation of SABV compliance mandatory in yearly progress reports and by using compliance as a contingency for grant renewals (both noncompetitive and competitive).” Actually, the way this is structured, combined with the following sentence about manuscript review, almost sidesteps the critical issue. I will not sidestep in this post.

We, peer scientists who are reviewing the grant proposals, are the ones who must take primary responsibility to assess whether a PI and associated Investigators have made a good faith attempt to follow/adopt SABV policy or not. Leaving this in the hands of Program to sort out, based on tepid review comments, is a dereliction of duty and will result in frustrating variablity of review that we all hate. So….we are the ones who will either let PIs off the hook, thereby undercutting everything NIH has tried to accomplish, or we will assist NIH by awarding poor scores to applications with a team that has not demonstrably taken SABV seriously. We are at a critical and tenuous point. Will PIs believe that their grants will still be funded with a carefully crafted SABV statement, regardless of whether they have followed through? Or will PIs believe that their grant getting is in serious jeopardy if they do not take the spirit of the SABV policy to heart? The only way this is decided is if the peer review scores reward those who take it seriously and punish those who do not.

So now we are back to the main point of this post which is how we are to assess good-faith efforts. I absolutely agree with Shansky and Murphy that an application (competing or not) that basically says “we’re going to follow up in the females later“, where later means “Oh we didn’t do it yet, but we pinky swear we will do it in this next interval of funding” should not be let off the hook.

However. What about a strategy that falls short of the “bare minimum”, as the authors insist on in Box 2, of including males and females in 50/50 proportion in every experiment, not powered to really confirm any potential sex difference?

I believe we need a little more flexibility in our consideration of whether the research of the PI is making a good faith effort or not. What I would like to see is simply that male and female studies are conducted within the same general research program. Sure, it can be the 50/50 group design. But it can also be that sometimes experiments are in males, sometimes in females. Particularly if there is no particular sense that one sex is always run first and the other is trivially “checked, or that one sex dominates the experimental rationale. Pubs might include both sexes within one paper, that’s the easiest call, but they might also appear as two separate publications. I think this can often be the right approach, personally.

This will require additional advocacy, thinking, pushback, etc, on one of the fundamental principles that many investigators have struggled with in the SABV era. As is detailed in Box 1 and 2 of the review, SABV does not mean that each study is a direct study of sex differences nor that every study in female mammals becomes a study of estrous cycle / ovarian hormones. My experience, as both an applicant and a reviewer, is that NIH study section members often have trouble with this notion. There has not been, in my experience on panels, a loud and general chorus rebutting any such notions during discussion either, we have much ground still to cover.

So we will definitely have to achieve greater agreement on what represents a good faith effort on SABV, I would argue, if we are to advocate strongly for NIH study sections to police SABV with the firm hand that it will require.

I object to what might be an obvious take-away from Shansky and Murphy, i.e., that the 50/50 sample approach is the essential minimum. I believe that other strategies and approaches to SABV can be done which both involve full single-sex sample sizes and do not require every study to be a direct contrast of the sexes in an experimentally clean manner.

BJP issues new policy on SABV

September 4, 2019

The British Journal of Pharmacology has been issuing a barrage of initiatives over the past few years that are intended to address numerous issues of scientific meta-concern including reproducibility, reliability and transparency of methods. The latest is an Editorial on how they will address current concerns about including sex as a biological variable.

Docherty et al. 2019 Sex: A change in our guidelines to authors to ensure that this is no longer an ignored experimental variable. https://doi.org/10.1111/bph.14761 [link]

I’ll skip over the blah-blah about why. This audience is up to speed on SABV issues. The critical parts are what they plan to do about it, with respect to future manuscripts submitted to their journal. tldr: They are going to shake the finger but fall woefully short of heavy threats or of prioritizing manuscripts that do a good job of inclusion.

From Section 4 BJP Policy: The British Journal of Pharmacology has decided to rectify this neglect of sex as a research variable, and we recommend that all future studies published in this journal should acknowledge consideration of the issue of sex. In the ideal scenario for in vivo studies, both sexes will be included in the experimental design. However, if the researcher’s view is that sex or gender is not relevant to the experimental question, then a statement providing a rationale for this view will be required.

Right? Already we see immense weaseling. What rationales will be acceptable? Will those rationales be applied consistently for all submissions? Or will this be yet another frustrating feature for authors in which our manuscripts appear to be rejected on grounds that other papers published seem to suffer from?

We acknowledge that the economics of investigating the influence of sex on experimental outcomes will be difficult until research grant‐funding agencies insist that researchers adapt their experimental designs, in order to accommodate sex as an experimental variable and provide the necessary resources. In the meantime, manuscripts based on studies that have used only one sex or gender will continue to be published in BJP. However, we will require authors to include a statement to justify a decision to study only one sex or gender.

Oh a statement. You know, the NIH has (sort of, weaselly) “insisted”. But as we know the research force is fighting back, insisting that we don’t have “necessary resources” and, several years into this policy, researchers are blithely presenting data at conferences with no mention of addressing SABV.

Overall sex differences and, more importantly, interactions between experimental interventions and sex (i.e., the effect of the intervention differs in the two sexes) cannot be inferred if males and females are studied in separate time frames.

Absolutely totally false. False, false, false. This has come up in more than one of my recent reviews and it is completely and utterly, hypocritically wrong. Why? Several reasons. First of all in my fields of study it is exceptionally rare that large, multi-group, multi-sub-study designs (in single sex) are conducted this way. It is resource intensive and generally unworkable. Many, many, many studies include comparisons across groups that were not run at the same time in some sort of cohort balancing design. And whaddaya know those studies often replicate with all sorts of variation across labs, not just across time within lab. In fact this is a strength. Second, in my fields of study, we refer to prior literature all the time in our Discussion sections to draw parallels and contrasts. In essentially zero cases do the authors simply throw up their hands and say “well since nobody has run studies at the same time and place as ours there is nothing worth saying about that prior literature”. You would be rightfully laughed out of town.

Third concern: It’s my old saw about “too many notes“. Critique without an actual reason is bullshit. In this case you have to say why you think the factor you don’t happen to like for Experimental Design 101 reasons (running studies in series instead of parallel) has contributed to the difference. If one of my peer labs says they did more or less the same methods this month compared to last year compared to five years ago…wherein lies the source of non-sex-related variance which explains why the female group self-administered more cocaine compared with the before, after and in between male groups which all did the same thing? And why are we so insistent about this for SABV and not for the series of studies in males that reference each other?

In conscious animal experiments, a potential confounder is that the response of interest might be affected by the close proximity of an animal of the opposite sex. We have no specific recommendation on how to deal with this, and it should be borne in mind that this situation will replicate their “real world.” We ask authors merely to consider whether or not males and females should be physically separated, to ensure that sight and smell are not an issue that could confound the results, and to report on how this was addressed when carrying out the study. Obviously, it would not be advisable to house males and females in different rooms because that would undermine the need for the animals to be exposed to the same environmental factors in a properly controlled experiment.

NO SHIT SHERLOCK!

Look, there are tradeoffs in this SABV business when it comes to behavior studies, and no doubt others. We have many sources of potential variance that could be misinterpreted as a relatively pure sex difference. We cannot address them all in each and every design. We can’t. You would have to run groups that were housed together, and not, in rooms together and not, at times similar and apart AND MULTIPLY THAT AGAINST EACH AND EVERY TREATMENT CONDITION YOU HAVE PLANNED FOR THE “REAL” STUDY.

Unless the objective of the study is specifically to investigate drug‐induced responses at specific stages of the oestrous cycle, we shall not require authors to record or report this information in this journal. This is not least because procedures to determine oestrous status are moderately stressful and an interaction between the stress response and stage of the oestrous cycle could affect the experimental outcome. However, authors should be aware that the stage of the oestrous cycle may affect response to drugs particularly in behavioural studies, as reported for actions of cocaine in rats and mice (Calipari et al., 2017; Nicolas et al., 2019).

Well done. Except why cite papers where there are oestrous differences without similarly citing cases where there are no oestrous differences? It sets up a bias that has the potential to undercut the more correct way they start Section 5.5.

My concern with all of this is not the general support for SABV. I like that. I am concerned first that it will be toothless in the sense that studies which include SABV will not be prioritized and some, not all, authors will be allowed to get away with thin rationales. This is not unique to BJP, I suspect the NIH is failing hard at this as well. And without incentives (easier acceptance of manuscripts, better grant odds) or punishments (auto rejects, grant triages) then behavior won’t change because the other incentives (faster movement on “real” effects and designs) will dominate.

SABV in NIH Grant Review

February 8, 2018

We’re several rounds of grant submission/review past the NIH’s demand that applications consider Sex As a Biological Variable (SABV). I have reviewed grants from the first round of this obligation until just recently and have observed a few things coming into focus. There’s still a lot of wiggle and uncertainty but I am seeing a few things emerge in my domains of grants that include vertebrate animals (mostly rodent models).

1) It is unwise to ignore SABV.

2) Inclusion of both sexes has to be done judiciously. If you put a sex comparison in the Aim or too prominently as a point of hypothesis testing you are going to get the full blast of sex-comparisons review. Which you want to avoid because you will get killed on the usual- power, estrus effects that “must” be there, various caveats about why male and female rats aren’t the same – behaviorally, pharmacokinetically, etc etc – regardless of what your preliminary data show.

3) The key is to include both sexes and say you will look at the data to see if there appears to be any difference. Then say the full examination will be a future direction or slightly modify the subsequent experiments.

4) Nobody seems to be fully embracing the SABV concept coming from the formal pronouncements about how you use sample sizes that are half males and half females into perpetuity if you don’t see a difference. I am not surprised. This is the hardest thing for me to accept personally and I know for certain sure manuscript reviewers won’t go for it either.

Then there comes the biggest categorical split in approach that I have noticed so far.

5a) Some people appear to use a few targeted female-including (yes, the vast majority still propose males as default and females as the SABV-satisfying extra) experiments to check main findings.

5b) The other take is just to basically double everything up and say “we’ll run full groups of males and females”. This is where it gets entertaining.

I have been talking about the fact that the R01 doesn’t pay for itself for some time now.
A full modular, $250K per year NIH grant doesn’t actually pay for itself.

the $250K full modular grant does not pay for itself. In the sense that there is a certain expectation of productivity, progress, etc on the part of study sections and Program that requires more contribution than can be afforded (especially when you put it in terms of 40 hr work weeks) within the budget.

The R01 still doesn’t pay for itself and reviewers are getting worse

I have reviewed multiple proposals recently that cannot be done. Literally. They cannot be accomplished for the price of the budget proposed. Nobody blinks an eye about this. They might talk about “feasibility” in the sense of scientific outcomes or preliminary data or, occasionally, some perceived deficit of the investigators/environment. But I have not heard a reviewer say “nice but there is no way this can be accomplished for $250K direct”.

Well, “we’re going to duplicate everything in females” as a response to the SABV initiative just administered the equivalent of HGH to this trend. There is approximately zero real world dealing with this in the majority of grants that slap in the females and from what I have seen no comment whatever from reviewers on feasibility. We are just entirely ignoring this.

What I am really looking forward to is the review of grants in about 3 years time. At that point we are going to start seeing competing continuation applications where the original promised to address SABV. In a more general sense, any app from a PI who has been funded in the post-SABV-requirement interval will also face a simple question.

Has the PI addressed SABV in his or her work? Have they taken it seriously, conducted the studies (prelim data?) and hopefully published some things (yes, even negative sex-comparisons)?

If not, we should, as reviewers, drop the hammer. No more vague hand wavy stuff like I am seeing in proposals now. The PI had better show some evidence of having tried.

What I predict, however, is more excuse making and more bad faith claims to look at females in the next funding interval.

Please prove me wrong, scientists in my fields of study.

__
Additional Reading:
NIH’s OER blog Open Mike on the SABV policies.
NIH Reviewer Guidance [PDF]

The first study section rounds that are obliged to grapple with the new SABV policy are upon us.
SROs are instructing panels and issuing grant assignments to reviewers.

If you are reviewing, what are your thoughts?

Me, I see more of the entirely predictable ahead- people ignoring it (or accepting thin excuses for not studying both sexes, in reality) or brandishing it as a cudgel in highly variable fashion. I’m cynical, perhaps unduly so, eh?

The opportunity to beat a panel into better agreement will come far too late for most applications. There is no way that guilt over consistency will drag the triaged apps up for discussion. 

I still seek consistency. In my own reviewing and in any panels I serve. I think this a virtue to strive for.

And I think that consideration and discussion of the approach to tricky review issues is the way to advance toward that goal. 

I also think that when you accept a reviewer position, you are agreeing to give the NIH what it is requesting, to the best of your ability. If you fight against the SABV push, you are doing it very wrong, IMO.

So….what do you think? How are you approaching the SABV mandate? Now that you have a few examples of how applicants have dealt with it, have you learned anything useful for us to consider?


Open Mike blog on SABV mandate

The NIH has recently issued the first round of guidance on inclusion of Sex as a Biological Variable in future NIH research grants. I am completely behind the spirit of the initiative but I have concerns about how well this is going to work in practice. I wrote a post in 2008 that detailed some of the reasons that have brought us to the situation where the Director of the NIH felt he had to coauthor an OpEd on this topic. I believe these issues are still present, will not be magically removed with new instructions to reviewers and need to be faced head-on if the NIH is to make any actual progress on ensuring SABV is considered appropriately going forward.

The post originally appeared December 2, 2008.


The title quote came from one of my early, and highly formative, experiences on study section. In the course of discussing a revised application it emerged that the prior version of the application had included a sex comparison. The PI had chosen to delete that part of the design in the revised application, prompting one of the experienced members of the panel to ask, quite rhetorically, “Why do they always drop the females?”

I was reminded of this when reading over Dr. Isis’ excellent post [Update: Original Sb post lost, I think the repost can be found here] on the, shall we say less pernicious, ways that the course of science is slanted toward doing male-based research. Really, go read that post before you continue here, it is a fantastic description.

What really motivated me, however, was a comment from the always insightful Stephanie Z:

Thank you. That’s the first time I’ve seen someone address the reasons behind ongoing gender disparities in health research. I still can’t say as it thrills me (or you, obviously), but I understand a bit better now.

Did somebody ring?

As I pointed out explicitly at least once ([Update: Original 2007 post]), research funding has a huge role in what science actually gets conducted. Huge. In my book this means that if one feels that an area of science is being systematically overlooked or minimized, one might want to take a close look at the manner by which science is funded and the way by which science careers are sustained as potential avenues for systematic remedy.

Funding

There are a couple of ways in which the generalized problems with NIH grant review lead to the rhetorical comment with which I opened the post. One very common StockCritique of NIH grant review is that of an “over ambitious” research plan. As nicely detailed in Isis’ post, the inclusion of a sex comparison doubles the groups right off the bat but even more to the point, it requires the inclusion of various hormonal cycling considerations. This can be as simple as requiring female subjects to be assessed at multiple points of an estrous cycle. It can be considerably more complicated, often requiring gonadectomy (at various developmental timepoints) and hormonal replacement (with dose-response designs, please) including all of the appropriate control groups / observations. Novel hormonal antagonists? Whoops, the model is not “well established” and needs to be “compared to the standard gonadectomy models”, LOL >sigh<.

manWomanControlPanel.jpg
Grant reviewers prefer simplicity
Keep in mind, if you will, that there is always a more fundamental comparison or question at the root of the project, such as “does this drug compound ameliorate cocaine addiction?” So all the gender comparisons, designs and groups need to be multiplied against the cocaine addiction/treatment conditions. Suppose it is one of those cocaine models that requires a month or more of training per group? Who is going to run all those animals ? How many operant boxes / hours are available? and at what cost? Trust me, the grant proposal is going to take fire for “scope of the project”.

Another StockCritique to blame is “feasibility”. Two points here really. First is the question of Preliminary Data- of course if you have to run more experimental conditions to establish that you might have a meritorious hypothesis, you are less likely to do it with a fixed amount of pilot/startup/leftover money. Better to work on preliminary data for two or three distinct applications over just one if you have the funds. Second aspect has to do with a given PIs experience with the models in question. More opportunity to say “The PI has no idea what s/he is doing methodologically” if s/he has no prior background with the experimental conditions, which are almost always the female-related ones. As we all know, it matters little that the hormonal assays or gonadectomy or whatever procedures have been published endlessly if you don’t have direct evidence that you can do it. Of course, more latitude is extended to the more-experienced investigator….but then s/he is less likely to jump into gender-comparisons in a sustained way in contrast to a newly minted PI.

Then there are the various things under grantspersonship. You have limited space in a given type of grant application. The more groups and comparisons, the more you have to squeeze in with respect to basic designs, methods and the interpretation/alternative approaches part. So of course you leave big windows for critiques of “hasn’t fully considered….” and “it is not entirely clear how the PI will do…” and “how the hypothesis will be evaluated has not been sufficiently detailed…”.

Career

Although research funding plays a huge role in career success, it is only part of the puzzle. Another critical factor is what we consider to be “great” or “exciting” science in our respective fields.

The little people can fill in the details. This is basically the approach of GlamourMagz science. (This is a paraphrase of something the most successful GlamourMagz PI I know actually says.) Cool, fast and hot is not compatible with the metastasizing of experimental conditions that is an inevitable feature of gender-comparison science. Trouble is, this approach tends to trickle down in various guises. Lower (than GlamourMag) impact factor journals sometimes try to upgrade by becoming more NS-like (Hi, J Neuro!). Meticulous science and exacting experimental designs are only respected (if at all) after the fact. Late(r) in someone’s career they start getting props on their grant reviews for this. Early? Well the person hasn’t yet shown the necessity and profit for the exhaustive designs and instead they just look…unproductive. Like they haven’t really shown anything yet.

As we all know splashy CNS pubs on the CV trump a sustained area of contribution in lower journals six ways to Sunday. This is not to say that nobody will appreciate the meticulous approach, they will. Just to say that high IF journal pubs will trump. Always.

So the smart young PI is going to stay away from those messy sex-differences studies. Everything tells her she should. If he does dip a toe, he’s more likely to pay a nasty career price.
This is why NIH efforts to promote sex-comparison studies are necessary. Promoting special funding opportunities are the only way to tip the equation even slightly more favorable to the sex-differences side. The lure of the RFA is enough to persuade the experienced PI to write in the female groups. To convince the new PI that she might just risk it this one time.

My suspicion is that it is not enough. Beyond the simple need to take a stepwise approach to the science as detailed by Isis, the career and funding pressures are irresistible forces.

The NIH has published NOT-OD-15-102 Consideration of Sex as a Biological Variable in NIH-funded Research which informs us:

This notice focuses on NIH’s expectation that scientists will account for the possible role of sex as a biological variable in vertebrate animal and human studies. Clarification of these expectations is reflected in plans by NIH’s Office of Extramural Research (OER) to update application instructions and review questions; once approved by the Office of Management and Budget (OMB), these updates will take effect for applications submitted for the January 25, 2016, due date and thereafter.

Also:

Accounting for sex as a biological variable begins with the development of research questions and study design. It also includes data collection and analysis of results, as well as reporting of findings. Consideration of sex may be critical to the interpretation, validation, and generalizability of research findings. Adequate consideration of both sexes in experiments and disaggregation of data by sex allows for sex-based comparisons and may inform clinical interventions. Appropriate analysis and transparent reporting of data by sex may therefore enhance the rigor and applicability of preclinical biomedical research.4

NIH expects that sex as a biological variable will be factored into research designs, analyses, and reporting in vertebrate animal and human studies. Strong justification from the scientific literature, preliminary data, or other relevant considerations must be provided for applications proposing to study only one sex. Investigators are strongly encouraged to discuss these issues with NIH program staff prior to submission of applications.

Additional information is provided in a three page PDF overview:

Literature review. Consider and describe how sex and gender may influence the research question(s) at hand. Conduct a review of the human clinical literature and any relevant preclinical literature. If there are differences between males and females in previous preclinical or clinical studies, this would provide a strong rationale for building consideration of sex into the research design and analyses of data. The absence of previous study data in an area of research does not, by itself, constitute strong justification to study only one sex.

Very nice. So helpful. Look NIH, clearly this is going to be a place where applicants who do not wish to incorporate SABV into the design are going to seek a loophole. What would be helpful here would be a more assertive statement about what does and, most importantly does not, constitute a “strong justification to study only one sex“. Uncontrolled, this will devolve back to the reviewers who are already failing (going by your highly effortful and high profile new initiative) to appropriately favor* SABV in research grant proposals. They are the ones that will decide that the tiniest fig leaf of excuse making is acceptable “justification” if you give them half a chance to do so. This part needs strengthening.

and later on in the document:

Single-sex studies. Applicants must provide strong justification for applications proposing to study only one sex. Such justification may include the study of sex-specific conditions or phenomena (e.g., ovarian or prostate cancer), acutely scarce resources (e.g., non-human primates), or investigations in which the study of one sex is scientifically appropriate. The absence of evidence regarding sex differences in an area of research does not constitute strong justification to study only one sex.

Sex-specific conditions or phenomena, check. Good. Will hard-to-breed mice constitute “acutely scarce resources”? Human drug abusers of various characteristics that make it hard to recruit female or male participants? The devil will be in the detail. But “scientifically appropriate“? Again, this holds open a big old loophole of escape. And a repeat of the absence-of-evidence statement. What does this mean? What are the limits on this strong justification? How are you going to get reviewers on board with this, instead of leaving them to accept any old excuse?

Research design, data analysis, and reporting.
….Where little or no sex-specific data is available, sex-specific hypotheses may not be possible, whereas previously observed sex differences may prompt sex-specific hypotheses.

Dude what? Are you kidding with this? We all know there must be a supported hypothesis in the research plan. And if there has not been any sex-differences research in the past, well, there are no hypotheses we can advance. And therefore, so sorry, we must avoid proposing anything that investigates SABV because the study section will kill us for lack of a clear hypothesis**. Another whoppingly huge escape clause for the SABV resistant PI.

Acknowledge limitations in the applicability of findings that may arise from the samples, methods, and analyses used, in the research plan as well as in progress reports and publications.

Emphasis added. HAHAHAHHAHHAA!!!! Yeah RIGHT! Every NIH Grant awardee who does not explicitly include SABV in a paper must make sure to add the caveat in the Discussion that their results cannot be extended to the other sex. Sure that’s going to happen. Sure.

Finally, one for my peers who already conduct SABV research with regularity.

Researchers working with animal models should consider if and how the female estrous cycle is relevant for experimental design and analysis; it may be relevant for some research questions and not others

This one is pointed straight at the buzz saw of the sex-differences aficionado Stock Criticism of grant applications. One of the ways that sex-differences gets stamped out of research proposals is that the “real” experts start in on “YOU MUST DO THE SEX-COMPARISONS RIGHT AND AS WE HAVE DONE”. This may include cycle synchronization, gonadectomy, pharmacologico-hormonal manipulations, endless groups, etc, etc, etc.

There is little tolerance from these people for “First, let’s give it a go in female (or male) animals/cells/tissues and see what we turn up” exploratory fishing expeditions.

I would argue that tolerance for fishing expeditions is precisely what the NIH needs if they want to jumpstart real change. You have to make the barrier low and, especially in this day and age, of low cost. Demanding that it has to be SABV design 101eleventy at all times or it is not worth doing is going to motivate resistance. Resistance on the part of PIs doing their grant proposing and on the part of peers doing the grant reviewing.

I propose that a NIH policy of “Any old Third Aim that will engage in sex-differences comparisons is good enough and a total freebie for the first five years***” is what is necessary.

_
*oh yes, believe you me there are puh-lenty of investigators who propose SABV aspects in proposals and get it beat out of them at review.

**StockCritiqueTM

***that may have to be slightly more formal

For some reason I am having a DOI error on the actual comment from Clayton and Collins. So until that is resolved, the sourcing is from the journalists who got the embargoed version.

Apparently Janine Clayton and Francis Collins have issued a commentary on a new policy that the NYT describes as:

The N.I.H. is directing scientists to perform their experiments with both female and male animals and include both sexes in sufficient numbers to see statistically significant differences. Grant reviewers will be instructed to take the sex balance of each study design into account when awarding grants.

Yeah, that sounds pretty clear. My studies just doubled…which means really that they were just cut in half. I’m cool with that. I actually agree that it would be good if we did almost everything as a sex-differences study.

There’s the money though. Sex difference studies in a behaving animal are not just a doubling as it happens (and as I inaccurately described it just above). From a prior post on this topic entitled: The funding is the science II, “Why do they always drop the females?

As nicely detailed in Isis’ post, the inclusion of a sex comparison doubles the groups right off the bat but even more to the point, it requires the inclusion of various hormonal cycling considerations. This can be as simple as requiring female subjects to be assessed at multiple points of an estrous cycle. It can be considerably more complicated, often requiring gonadectomy (at various developmental timepoints) and hormonal replacement (with dose-response designs, please) including all of the appropriate control groups / observations. Novel hormonal antagonists? Whoops, the model is not “well established” and needs to be “compared to the standard gonadectomy models”, LOL >sigh<.

The money and the progress.

Keep in mind, if you will, that there is always a more fundamental comparison or question at the root of the project, such as “does this drug compound ameliorate cocaine addiction?” So all the gender comparisons, designs and groups need to be multiplied against the cocaine addiction/treatment conditions. Suppose it is one of those cocaine models that requires a month or more of training per group? Who is going to run all those animals ? How many operant boxes / hours are available? and at what cost?

Oh, don’t worry bench jockeys. According to the NYT article:

Researchers who work with cell cultures are also being encouraged to study cells derived from females as well as males, and to do separate analyses to tease out sex differences at the cellular level.

“Every cell has a sex,” Dr. Clayton said in a telephone interview. “Each cell is either male or female, and that genetic difference results in different biochemical processes within those cells.”

“If you don’t know that and put all of the cells together, you’re missing out, and you may also be misinterpreting your data,” Dr. Clayton added. For example, researchers recently discovered that neurons cultured from males are more susceptible to death from starvation than those from females, because differences in the ways their cells process nutrients.

“Encouraged”. Okay, maybe you CultureClowns have an escape clause here. Animal model folks are facing “demanded” language.

Final observations are ridiculous:

But [the new policies] are likely to be met with resistance from scientists who fear increased costs and difficulty in performing their experiments. Studying animals of both sexes may potentially double the number required in order to get significant results.

“There’s incredible inertia among people when it comes to change, and the vast majority of people doing biological research are going to think this is a huge inconvenience,” Dr. Zucker said.

Margaret McCarthy, a neuroscientist at University of Maryland School of Medicine who studies sex differences, agreed. “The reactions will range from hostile — ‘You can’t make me do that’ — to ‘Oh, I don’t want to control for the estrous cycle,’” she said.

This has nothing to do with whether a scientist “wants” to or not.

Let me be clear, I want to do sex-differences studies. I am delighted that this will be a new prescription. I agree with the motivating sentiments.

What I “fear” is that grant applications will be kicked in the teeth if they include sex differences comparisons. What I “fear” is that my research programs will be even less productive on the main area of interest, to the tune of a lot of extra work that will simply confirm a lot of what we already know. For example, female rats tend to self-administer more drug than males do. A lot of my colleagues have been working on these topics for a long time. The identification of those areas where it actually matters (i.e., sex difference effects that haven’t yet been detected) are going to come along with a lot of negative findings. What I “fear” is that when we are interested in a certain thing, there is a bit of sex-differences literature and the hypothesis is going to be “males and females are the same” or even “females are more/less sensitive to drug” that this is going to bring down the holy hells of reviewer wrath over what hypothesis we are testing.

I fear a lot of things about this. What I don’t fear is my own interest in the topic. What I don’t fear is the “inconvenience”. I don’t even fear “difficulty”. It just isn’t that difficult to add female groups to my studies.

What it takes is additional grant funding. Or tolerance on the part of P&T committees, hiring committees and grant review panels for apparently reduced progress on a scientific topic of interest. And those things are not at all easy to come by.

The funny thing is, we’ve been taking steps in the lab toward this direction in the past year anyway. So I should be grateful I have at least that little tiny bit of a head start on this stuff.