You know the old story.

In this new story, we have the NIH’s Sex As a Biological Variable (SABV) policy. When first discussed, just about everyone who took this seriously pointed out the problem of a zero sum, limited funding system adopting a mandate which would double the animal costs. To really consider SABV properly, we said, this is going to double our sample sizes…at the very least. Probably more than double.

That is coming from the perspective of a scientist who works with units of the whole experimental animal. There are many of us.

The official NIH response was a bunch of gaslighting.

“Oh no”, went the policy mavens of the NIH, “this is not what this means at all. Simply include equal numbers of male and female animals at your regular sample size. That’s it. Oh, yeah, you have to say you will stratify your data by sex and look at it. You know, just in case there’s anything there. But nothing insists you have to double your sample size.”

Sure, said we NIH watchers/applicants. Sure it will go like that. Have you met our reviewers? They are going to first of all demand that every study is fully powered to detect any sex difference. Then, they are going to immediately start banging on about swabbing and cycling the female rats and something something about powering up for cycle as well.

NIH: “No, of course not that would never happen why we will tell them not to do that and everything will be copacetic”

Things were not copacetic. As predicted, reviewers of grants have, since even before the mandate went into effect, demonstrated they are constitutionally unable to do what NIH claimed they should be doing and in fact do what they were predicted in advance to do. Make everything HAVE to be a sex differences study and HAVE to be a study of estrous cycle. Randomly. Variable. Yes. As with everything in NIH review. And who knows, maybe this is a selective cudgel (I call it Becca’s Bludgeon) used only when they just generally dislike the proposal.

The NIH mandate let the SABV camel’s nose under the tentflap and now that camel is puuuuuuuuussssshhhhing all the way in.

A new article in eLife by Garcia-Sifuentes and Maney is part of this campaign. It is chock full of insinuations and claims trying to justify the full camel in side the tent. Oh, they know perfectly well what the NIH policy was. But they are using all of the best #allegedprofession techniques to try to avoid admitting they are fully doing an end run.

From the Abstract: This new policy has been interpreted by some as a call to compare males and females with each other.

From the Intro: Although the NIH policy does not explicitly require that males and females be compared directly
with each other, the fact that more NIH-funded researchers must now study both sexes should lead to an increase in the frequency of such comparisons (insert self-citation). For example, there should be more testing for sex-specific
responses

“should”.

although the proportion of articles that included both sexes significantly increased (see also Will et al., 2017), the proportion that treated sex as a variable did not. [Note interesting goalpost move. or at least totally undefined insinuation] This finding contrasts sharply with expectations [whose “expectations” would those be?], given not only the NIH mandate but also numerous calls over the past decade to disaggregate all preclinical data by sex [yes, the mandate was to disaggregate by sex. correct.] and to test for sex differences [bzzzt, nope. here’s another slippery and dishonest little conflation]

One potential barrier to SABV implementation is a lack of relevant resources; for example, not all researchers have received training in experimental design and data analysis that would allow them to test for sex differences using appropriate statistical approaches. [oh what horseshit. sure, maybe there is a terrible lack of experimental design training. I agree those not trained in experimental psychology seem to be a bit lacking. But this is not specific to sex differences. A group is a group is a group. so is a factor. the “lack of relevant resources” is….money. grant money.]

any less-than-rigorous test for sex differences creates risk for misinterpretation of results and dissemination of misinformation to other scientists and to the public [There you have it. The entire NIH scheme to introduce SABV is not only flawed, it is, seemingly, even worse than doing nothing!]

Although a sex difference was claimed in a majority of articles (57%), not all of these differences were supported with statistical evidence. In more than a quarter of the articles reporting a sex difference, or 24/83 articles, the sexes were never actually compared statistically. [Yep, totally consistent with the assertions from NIH about what they were after. Anything else is a significant move of the goalposts. In the direction that was anticipated and EXPLICITLY denied as being the goal/end game by the NIH. In oh so many ways.]

In these cases, the authors claimed that the sexes responded differentially to a treatment when the effect of treatment was not statistically compared across sex. … Of the studies with a factorial design, 58% reported that the sexes responded differently to one or more other factors. The language used to state these conclusions often included the phrase ‘sex difference’ but could also include ‘sex-specific effect’ or that a treatment had an effect ‘in males but not females’ or vice versa. … Neither approach tests whether the treatment had different effects in females and males. Thus, a substantial majority of articles containing claims of sex-specific effects (70%) did not present statistical evidence to support those claims

This is also, utter a-scientific horseshit.

I get this a lot from reviewers so I’m going to expand but only briefly. There is no such thing as canonical statistical interpretation techniques that are either “right” or “wrong”. Nor do statistical inference techniques alter the outcome of a study. The data are what they are. All else is shades of interpretation. At the very best you could say that different inferential statistical outcomes may mean there is stronger or weaker evidence for your interpretations of the data. at best.

But there is a broader hypocrisy here. Do you only build your knowledge within the context of one paper? Do you assemble your head space on whether something is likely or unlikely to be a valid assertion (say, “female rats self-administer more cocaine”) ONLY on papers that provide like-to-like perfectly parallel and statistically compared groups?

If you are an idiot, I suppose. Not being an idiot, I assert that most scientists build their opinions about the world of science that they inhabit on a pile of indirectly converging evidence. Taking variability in approach into account. Stratifying the strength of the evidence to their best ability. Weighting the results. Adding each new bit of evidence as they come across it.

And, in a scenario where 10 labs were conducting cocaine self-administration studies and five each tended to work on males and females independently, we would conclude some things. If we were not preening Experimental Design Spherical Cow 101 idiots. If, for example, no matter the differences in approach it appeared that in aggregate the females self-administered twice as many infusions of the cocaine.

We would consider this useful, valid information that gives us the tentative idea that perhaps there is a sex difference. We would not hold our hands over our eyes mumbling “blah blah blah I can’t hear you either” and insist that there is zero useful indication from this true fact. We would, however, as we do with literally every dataset, keep in mind the limitations of our inferences. We might even use these prior results to justify a better test of the newly developed hypothesis, to overcome some of the limitations.

That is how we build knowledge.

Not by insisting if a comparison of datasets/findings does not accord with strict ideas of experimental design rigor, it is totally invalid and meaningless.

Among the articles in which the sexes were pooled, the authors did so without testing for a sex difference almost half of the time (48%; Figure 3B). When authors did test for a sex difference before pooling, they sometimes found a significant difference yet pooled the sexes anyway; this occurred in 17% of the articles that pooled.[Yes, consistent with the NIH policy. Again with the moving the goalposts….]

Thus, the authors that complied with NIH guidelines to disaggregate data usually went beyond NIH guidelines to explicitly compare the sexes with each other. [hookay…..so where’s the problem? isn’t this a good thing?]

Asked and Answered

June 2, 2021

A tweet in response to a question I asked

said that perhaps a grad student’s job is to learn to answer questions and a postdoc’s job is to learn to ask questions.

I thought about that for half a second and concluded that this is backwards, for me. I think that I started into grad school thinking I knew how to ask scientific questions. I then spent the entirety of my time in grad school, right up until my committee signed off on my dissertation, learning the hard way that this was not so. I concluded that the main part of my graduate school training was learning how (not) to ask scientifically tractable questions.

In my postdoctoral training, I think that I learned how to answer questions. Not in the limited sense of “conduct this experiment, analyze the data and conclude a result”. Answering a question in the much broader sense of deploying available resources to address a scientifically tractable question and to bring this to an “answer” that was publishable in the scientific record.

I believe my career as a PI simply extends upon this, in the sense that my job is to secure the available resources in a broader sense and that “tractable” now includes the ability to direct the hands of more people. And the questions may no longer be my questions, but rather the questions of those who are in my laboratory. But it’s all just answering questions.

Rule Followers

February 12, 2021

As always Dear Reader, I start with personal confession so you know how to read my biases appropriately.

I am a life time Rule Follower.

I am also a life time Self-Appointed Punisher of Those Who Think the Rules Do Not Apply to Them.

What does “Rule Follower” mean to me? No, not some sort of retentive allegiance to any possible guideline or rule, explicit or implicit. I’ve been known to speed once in awhile. It doesn’t even mean that rule followers are going to agree with, and follow, every rule imaginable for any scenario. It is just an orientation of a person that believes there are such things as rules of behavior, these rules are good things as a social or community compact and that it is a good idea to adhere to them as a general rule. It is a good idea to work within the rules and that this is what is best for society, but also for the self.

The other kind of person, the “Rules Don’t Apply to ME” type, is not necessarily a complete sociopath*. And, in fact, such people may actually be a Rule Follower when it comes to the really big, obvious and Important (in their view) rules. But these are people that do not agree that all of the implicit social rules that Rule Followers follow actually exist. They do not believe that these rules apply to them, and often extend that to the misdemeanor sort of actual formal Rules, aka The Law.

Let’s talk rules of the road- these are the people who routinely speed, California Stop right on reds, and arc into the far lane when making a turn on a multi-lane road. These are the people that bypass a line of patiently waiting traffic and then expect to squeeze into the front of the line with an airy “oops, my badeee, thanks!” smile and wave. They are the ones that cause all sorts of merging havoc because they can’t be arsed to simply go down to the next street or exit to recover from their failure to plan ahead. These are often the people who, despite living in a State with very well defined rules of the road for bicycle traffic, self-righteously violate those rules as a car driver and complain about how the law-compliant bicycle rider is the one in the wrong.

But above all else, these people feel entitled to their behavior. It is an EXTREME OUTRAGE whenever they are disciplined in any way for their selfish and rude behavior that is designed to advantage themselves at the cost to (many) others.

If you don’t let them in the traffic line, you are the asshole. When you make the left turn into lane 2 and they barely manage to keep from hitting you as they fail to arc their own turn properly..you are the asshole. When they walk at you three abreast on the sidewalk and you eyeball the muppethugger trying to edge you off your single lane coming the other way and give every indication you are willing to bodycheck their selfish ass until they finally grudgingly rack it the fuck in…YOU are the asshole.

When they finally get a minor traffic citation for their speeding or failing to stop on a right on red… Oh, sister. It’s forty minutes of complaining rationalization about how unfair this is and why are those cops not solving real crimes and oh woe is me for a ticket they can easily pay. Back in the day when it was still illegal, this was the person caught for a minor weed possession citation who didn’t just pay it but had to go on at length about how outrageous it was to get penalized for their obvious violation of the rules. Don’t even get me started about how these people react to a citation for riding their bicycle on the sidewalk (illegal!) instead of in the street (the law they personally disagree with).

Back before Covid you could identify these two types by hanging around the bulk food bin at your local hippy grocery store. Rule Followers do not sample the items before paying and exiting the store. Those other people…..

Hopefully I’ve chosen examples that get you into the proper mindset of a complex interplay of formal rules that not everyone follows and informal rules of conduct that not everyone follows. I shouldn’t have to draw your attention to how the “Rules Don’t Apply to Me” sail along with convenient interpretations, feigned ignorances and post-hoc everyone-does-it rationales to make their lives a lot easier. That’s right, it’s convenient to not follow the rules, it gets them ahead and frankly those Rule Followers are beta luser cucks for not living a life of personal freedom!

We’re actually in the midst of one of these scenarios right now.

Covid vaccination

As you are aware, there are formal “tiers” being promulgated for who gets schedule for vaccines at which particular time. You know the age cutoffs- we started with 75+ and are now at 65+ in most locations. Then there are the job categories. Health care workers are up first, and then we are working a cascade of importance given occupation. Well, in my environment we had a moment in which “lab workers” were greenlit and Oh, the science lab types rushed to make their appointments. After a short interval, the hammer came down because “lab” meant “lab actually dealing with clinical care and health assessment samples” and not just “any goofaloon who says they work in a lab”.

Trust me, those at the head of that rush (or those pushing as the lab head or institution head) were not the Rule Followers. It was, rather, those types of people who are keen to conveniently define some situation to their own advantage and never consider for a second if they are breaking the Rules.

Then there have been some vaccine situations that are even murkier. We’ve seen on biomedical science tweeter that many lab head prof types have had the opportunity to get vaccinated out of their apparent tier. It seemed, especially in the earlier days prior to vaccine super centers, that a University associated health system would reach the end of their scheduled patients for the day and have extra vaccine.

[ In case anyone has been hiding under a rock, the first vaccines are fragile. They have to be frozen for storage in many cases and thus thawed out. They may not be stable overnight once the vial in question has been opened. In some cases the stored version may need to be “made up” with vehicles or adjuvants or whatever additional components. ]

“Extra” vaccine in the sense of active doses that would otherwise be lost / disposed of if there was no arm to stick it in. Employees who are on campus or close by, can readily be rounded up on short notice, and have no reason to complain if they can’t get vaccinated that particular day, make up this population of arms.

Some Rule Followers were uncomfortable with this.

You will recognize those other types. They were the ones triumphantly posting their good luck on the internet.

In my region, we next started to have vaccine “super centers”. These centers recruited lay volunteers to help out, keep an eye on patients, assist with traffic flow, run to the gloves/syringe depot, etc. And, as with the original health center scenario, there were excess doses available at the end of the day which were offered to the volunteers.

Again, some Rule Followers were uncomfortable with this. Especially because in the early days it was totally on the DL. The charge nurse closest to you would pull a volunteer aside and quietly suggest waiting around at the end of the day just “in case”. It was all pretty sketchy sounding….. to a Rule Follower. The other type of person? NO PROBLEM! They were right there on day one, baby! Vacc’d!

Eventually the volunteer offer policy became someone formalized in my location. Let me tell you, this was a slight relief to a Rule Follower. It for sure decreases the discomfort over admitting one’s good fortune on the intertoobs.

But! It’s not over yet! I mean, these are not formalized processes and the whole vaccine super-center is already chaos just running the patients through. So again, the Rules Don’t Need To Be Followed types are most likely to do the self-advocacy necessary to get that shot in their arm as quickly and assuredly as possible. Remember, it’s only the excess doses that might be available. And you have to keep your head up on what the (rapidly shifting and evolving) procedure might be at your location if you want to be offered vaccine.

Fam, I’m not going to lie. I leaned in hard on anyone I think of as a Rule Follower when I was relating the advantages of volunteering** at one of our vaccine super-centers. I know what we are like. I tell them as much about the chaotic process as I know so as to prepare them for self-advocacy, instead of their native reticence to act without clear understanding of rules that entitle them to get stuck with mRNA.

Still with me?

NIH has been cracking down on URLs in grant applications lately. I don’t know why and maybe it has to do with their recent hoopla about “integrity of review” and people supposedly sharing review materials with outside parties (in clear violation of the review confidentiality RULES, I will note). Anyway, ever since forever you are not supposed to put URL links in your grant applications and reviewers are exhorted never ever to click on a link in a grant. It’s always been explained to me in the context of IP address tracking and identifying the specific reviewers on a panel that might be assigned to a particular application. Whatever. It always seemed a little paranoid to me. But the Rules were exceptionally clear. This was even reinforced with the new Biosketch format that motivated some sort of easy link to one’s fuller set of publications. NIH permits PubMed links and even invented up this whole MyBibliography dealio at MyNCBI to serve this purpose.

Anyway there has been a few kerfuffles of EXTREME ANGER on Science Twitter from applicants who had their proposals rejected prior to review for including URLs. It is an OUTRAGE, you see, that they should be busted for this clear violation of the rules. Which allegedly, according to Those To Whom Rules Do Not Apply, were incredibly arcane rules that they could not possibly be expected to know and waaah, the last three proposals had the same link and weren’t rejected and it isn’t FAAAAAIIIIR!

My gut reaction is really no different than the one I have turning left in a two lane turn or walking at sidewalk hogs. Or the one I have when a habitual traffic law violator finally has to pay a minor fine. Ya fucked around and found out. As the kids say these days.

For some additional perspective, I’ve been reviewing NIH grants since the days when paper hard copies were submitted by the applicant and delivered to the reviewers as such. Pages could be missing if the copier effed up- there was no opportunity to fix this once a reviewer noticed it one week prior to the meeting. Font size shenanigans were seemingly more readily played. And even in the days since, as we’ve moved to electronic documents, there are oodles and oodles of rules for constructing the application. No “in prep” citations in the old Biosketch….people did it anyway. No substituting key methods in the Vertebrate Animals section…..people still do it anyway. Fonts and font size, okay, but what about vertical line spacing….people fudge that anyway. Expand figure “legends” (where font size can be smaller) to incorporate stuff that (maybe?) should really be in the font-controlled parts of the text. Etc, etc, etc.

And I am here to tell you that in many of these cases there was no formal enforcement mechanism. Ask the SRO about a flagrant violation and you’d get some sort of pablum about “well, you are not obliged to consider that material..”. Font size? “well…..I guess that’s up to the panel”. Which is enraging to a Rule Follower. Because even if you want to enforce the rules, how do you do it? How do you “ignore” that manuscript described as in prep, or make sure the other reviewers do? How do you fight with other reviewers about how key methods are “missing” when they are free to give good scores even if that material didn’t appear anywhere in figure legend, Vertebrate Animals or, ISYN, a 25% of the page “footnote” in microfont. Or how do your respond if they say “well, I’m confident this investigator can work it out”?

If, in the old days, you gave a crappy score to a proposal that everyone loved by saying “I put a ruler on the vertical and they’ve cheated” the panel would side eye you, vote a fundable score and fuck over any of your subsequent proposals that they read.

Or such might be your concern if your instinct was to Enforce the Rules.

Anyway, I’m happy to see CSR Receipt and Referral enforce rules of the road. I don’t think it an outrage at all. The greater outrage is all the people who have been able to skirt or ignore the rules and advantage themselves against those of us who do follow the rules***.

__

*Some of my best friends are habitual non-followers-of-rules.

**I recommend volunteering at a vaccine super station if you have the opportunity. It is pretty cool just to see how your health care community is reacting in this highly unusual once-in-a-generation crisis. And its cool, for those of us with zero relevant skills, to have at least a tiny chance to help out. Those are the Rules, you know? 🙂

***Cue Non-Followers-of-Rules who, Trumplipublican- and bothsiders-media-like, are absolutely insistent then when they manage to catch a habitual Rule Follower in some violation it proves that we’re all the same. That their flagrant and continual behavior is somehow balanced by one transgression of someone else.

It’s Uninterpretable!

August 6, 2020

No, it isn’t.

One of my favorite species of manuscript reviewer comment is that the data we are presenting are “uninterpretable”. Favorite as in the sort of reaction I get where I can’t believe my colleagues in science are this unbelievably stupid and are not completely embarrassed to say any such thing ever.

“Uninterpretable” is supposed to be some sort of easy-out Stock Critique, I do understand that. But it reveals either flagrant hypocrisy (i.e., the reviewer themselves would fall afoul of such a criticism with frequency) or serious, serious misunderstanding of how to do science.

Dr. Zen is the latest to run afoul of my opinion on this. He posted a Tweet:

and then made the mistake of bringing up the U word.

(his followup blog post is here)

Now, generally when I am laughing at a reviewer comment, it is not that they are using “uninterpretable” to complain about graphical design (although this occasionally comes into the mix). They usually mean they don’t like the design of the experiment(s) in some way and want the experiment conducted in some other way. Or the data analyzed in some other way (including graphical design issues here) OR, most frequently, a whole bunch of additional experiments.


“If the authors don’t do this then the data they are presenting are uninterpretable” – Reviewer # 3. It’s always reviewer #3.

Let me address Zen’s comment first. It’s ridiculous. Of COURSE the graph he presented is interpretable. It’s just that we have a few unknowns and some trust. A whole lot of trust. And if we’ve lost that, science doesn’t work. It just doesn’t. So it’s ridiculous to talk about the case where we can’t trust that the authors aren’t trying to flagrantly disregard norms and to lie to us with fake data. There’s just no point. Oh and don’t forget that Zen construed this in the context of a slide presentation. There just isn’t time for minutia and proving beyond any doubt that the presenter/authors aren’t trying to mislead with fakery.

Scientific communication assumes some reasonable common ground, particularly within a subfield. This is okay. When there is cross talk between fields with really, really different practices, ok, maybe a little extra effort is needed.

But this is a graph using the box-and-whiskers plot. This is familiar to the audience and indeed Zen does not seem to take issue with it. He is complaining about the exact nature of the descriptive statistic conventions in this particular box-and-whiskers plot. He is claiming that if this is not specified that the data are “uninterpretable”. NONSENSE!

These plots feature an indicator of central tendency of a distribution of observations, and an indicator of variablity in that distribution. Actually, most descriptive illustrations in science tackle this task. So..it’s familiar. This particular type of chart gives two indications of the variability- a big one and a small one. This is baseline knowledge about the chart type and, again, is not the subject of Zen’s apparent ire. The line is the central tendency. The box outlines the small indicator and the whiskers outline the big indicator. From this we move into interpretation that is based on expectations. Which are totally valid to deploy within a subfield.

So if I saw this chart, I’d assume it was most likely depicting the central tendency of a median or mean. Most likely the median, particularly if the little dot indicates the mean. The box therefore outlines the intraquartile range, i.e., the 25%ile and 75%ile values. If the central tendency is the mean, then it is most likely that the box outlines plus or minus one standard error of the mean or one standard deviation. Then we come to the whiskers. I’d assume it was either the 95% Confidence Interval or the range of values.

I do NOT need to know which of these minor variants is involved to “interpret” the data. Because scientific interpretation functions along a spectrum of confidence in the interpretation. And if differences between distributions (aha another ready assumption about this chart) cannot be approximated from the presentation then, well, it’s okay to delve deeper. To turn to the inferential statistics. In terms of if the small indicator is SD or SEM? meh, we can get a pretty fair idea. If it isn’t the SD or SEM around a mean, or the 25%ile/75%ile around a median, but something else like 3SEM or 35/65? Well, someone is doing some weird stuff trying to mislead the audience or is from an entirely disparate field. The latter should be clear.

Now, of COURSE, different fields might have different practices and expectations. Maybe it is common to use 5 standard deviations as one of the indicators of variability. Maybe it is common to depict the mode as the indicator of central tendency. But again, the audience and the presenter are presumably operating in approximately the same space and any minor variations in what is being depicted do not render the chart completely uninterpretable!

This is not really any different when a manuscript is being reviewed and the reviewers cry “Uninterpretable!”. Any scientific paper can only say, in essence, “Under these conditions, this is what happened”. And as long as it was clear what was done and the nature of the data, the reporting of can be interpreted. We may have more or fewer caveats. We may have a greater or smaller space of uncertainty. But we can most certainly interpret.

It sometimes gets even worse and more hilarious. I have this common area where we present data where the error bars are smaller than the (reasonably sized) symbols for some (but not all) of the groups. And we may have cases where the not-different (by inferential stats *and* by any rational eyeball and consideration of the data at hand) samples cannot be readily distinguished from each other (think: overlapping longitudinal or dose curves).

“You need to use color or something else so that we can see the overlapping details or else it is all uninterpretable!” – Reviewer 3.

My position is that if the eye cannot distinguish any differences this is the best depiction of the data. What is an error is presenting data in a way that gives some sort of artificial credence to a difference that is not actually there based on the stats, the effect size and a rational understanding of the data being collected.

I would have predicted, if anyone had told me a few months back that I’d be sitting under home quarantine for weeks at a time, that I’d be blogging up a storm to compensate.

Obviously, I’m not.

Our business of doing science has taken a serious shot right in the fo’castle and we, most of us anyway, are not doing things the same on a day to day basis. You would think I would have things to say about this. And maybe I do, I just have no idea where to begin.

I’m scared for my lab’s survival. I almost always am, true, but this is different. I’m not going to sing you my tale of woes today because many, many of you are in the same boat.

The shut down doesn’t do much of anything good for our usual problems and anxieties. There has been some relief for the tenure seekers, true. Many Universities have announced that there will be tenure clock delays permitted and that everybody in the process, from Tenure and Promotions Committees to letter writers will be exhorted to take the Time of Corona into account when assessing productivity. There has been some relief for those who are paid from NIH grants in that NIH has basically said it is okay to keep paying people even if their productivity has changed dramatically.

But this doesn’t help the person who is seeking tenure to actually get tenure, as far as I can tell. It’s not as if a delay in clock makes things magically better. We often have years-long arcs of developing our research programs, of making the efforts of our laboratories pay off in published work. And while yes, if you happened to be doing well prior to March of this year, you can ride that. But if you were just getting going? If the research models were finally reaching productivity? If maybe you had just managed to secure a grant, at long last, and were looking to CRANK it for a year to ensure tenure? Or maybe you were just about to collect that preliminary data that was going to push your 12th R01 attempt over the line…?

How is there any predictable way that a delayed clock or supposed relaxation of review standards are supposed to help with this? Unless the assurance from your University is that they are just going to hand you tenure now, I’m sorry, but you should be terrified. I am terrified FOR you.

Grants. Ah, grants. Yes the NIH has reiterated there is spending flexibility. But all we are doing is burning daylight. Staff are being paid but we’re getting less productivity per person hour. If we are doing it right, that is. If we are, in fact, shutting it down. Those weeks and months are just ticking away. And we are still in the same nightmare of funding….only worse. There is no guarantee that grant review in the coming rounds will take Corona-related excuses seriously. And even if they do, this is still competition. A competition where if you’ve happened to be more productive than the next person, your chances are better. Are the preliminary data supportive? Is your productivity coming along? Well, the next PI looks fine and you look bad so…. so sorry, ND. Nobody can ever have confidence that where they are when they shut down for corona will ever be enough to get them their next bit of funding.

I don’t see any way for the NIH to navigate this. Sure, they could give out supplements to existing grants. But, that only benefits the currently funded. Bridge awards for those that had near-miss scores? Sure, but how many can they afford? What impact would this have on new grants? After all, the NIH shows no signs yet of shutting down receipt and review or of funding per Council round as normal. But if we are relying on this, then we are under huge pressure to keep submitting grants as normal. Which would be helped by new Preliminary Data. And more publications.

So we PIs are hugely, hugely still motivated to work as normal. To seek any excuse as to why our ongoing studies are absolutely essential. To keep valuable stuff going, by hook or by crook….

Among other reasons, WE DON’T KNOW THE END DATE!

It could be olly-olly ox in free at almost any moment. If we get relieved from these duck-and-cover restrictions in a week or two, well, those who euthanized a bunch of research subjects are going to look really, really stupid. If we battened down the lab for the six-month window, we’re going to be a lot slower to get back up to speed. And those June/July grant submission dates are fast approaching. So are the October / November ones, frankly.

I have no answers. I know for a fact that some folks are fighting lab closures inch by inch and continue to generate data. I know some other folks shut it right down to zero at the first intimation this was coming. I know the former will be advantaged in the very near future and the latter will pay a price.

and winter is coming.

A semi-thread from frustrated bioinformaticians emerged on twitter recently. In it they take shots at their (presumably) collaborators who do not take their requests for carefully curated and formatted data to heart.

Naturally this led me to taunt the data leech OpenScienceEleventy waccaloons for a little bit. The context is probably a little different (i.e., it seems to reference established collaborations between data-generating and data-analyzing folks) but the idea taps on one of my problems with the OpenScience folks. They inevitably don’t just mean they want access to the data that went into your paper but ALL of your data related to it. Down to the least little recorded unit (someone in the fighty thread said he wanted raw electrophysiological recording to test out his own scoring algorithm or some such). And of course they always mean that it should be nicely formatted in their favorite way, curated for easy understanding by computer (preferably) and, in all ways, the burden should be on the data-generating side to facilitate easy computational analysis. This is one of the main parts that I object to in their cult/movement- data curation in this way comes with a not-insubstantial cost expended to the benefit of some internet random. I also object on the basis of the ownership issues, bad actors (think: anti-science extremists of various stripes including right wing “think tanks” and left wing animal right terrorists), academic credit, opportunity loss among other factors.

However, the thought of the day is about data curation and how it affects the laboratory business and my mentoring of science trainees. I will declare that consistent data collation, curation, archiving and notation is a good thing for me and for my lab. It helps the science advance. However, these things come at a cost. And above all else when we consider these things, we have to remember that not every data point collected enters a scientific manuscript or is of much value five or ten years down the line. Which means that we are not just talking about the efficient expenditure of effort on the most eventually useful data, we’re talking about everything. Does every single study get the full data analysis, graphical depiction and writeup? Not in my lab. Data are used at need. Data are curated to the extent that it makes sense and sometimes that is less than complete.

Data are collected in slightly different ways over time. Maybe we changed the collection software. Maybe our experiments are similar, but have a bit of a tweak to them. Maybe the analyses that we didn’t think up until later might be profitably applied to earlier datasets but…..the upside isn’t huge compared to other tasks. Does this mean we have to go back and re-do the prior analyses with the current approach? If we want to, this sometimes that requires that third and fourth techniques (programs, analysis strategies, etc) be created and applied. This comes with additional effort costs. So why would we expend those efforts for something? If there was interest or need on the part of some member of the laboratory, sure. If a collaborator “needs” that analysis, well, this is going to be case by case on the basis of what it gains us, the collaboration or maybe the funded projects. Because it all costs. Time, which is money, and the opportunity cost of those staff members (and me) not doing other tasks.

Staff members. Ah, yes, the trainees. I am totally supportive of academic trainees who want to analyze data and come up with new ways to work with our various stock-in-trade data sets and archive of files. This, btw, is what I did at one of my postdoctoral stops. I was working with a model where we were somewhat captive to the rudimentary data analyses provided by the vendor’s software. The data files were essentially undocumented, save for the configuration data, dates and subject identifiers. I was interested in parsing the data in some new ways so I spent a lot of time making it possible to do so. For the current files I was collecting and for the archive of data collected prior to my arrival and for the data being collected by my fellow trainees. In short, I faced the kind of database that OpenData people claim is all they are asking for. Oh, just give us whatever you have, it’s better to have anything even if not annotated, they will claim. (Seriously). Well, I did the work. I was able to figure out the data structure in the un-annotated files. This was only possible because I knew how the programs were working, how the variables could be set for different things, what the animals were doing in a general sense in terms of possible responses and patterns, how the vendor’s superficial analysis was working (for validation), what errors or truncated files might exist, etc. I wrote some code to create the slightly-more-sophisticated analyses that I happened to dream up at the time. I then started on the task of porting my analysis to the rest of the lab. So that everyone from tech to postdoc was doing initial analysis using my programs, not the vendor ones. And then working that into the spreadsheet and graphing part of the data curation. And THEN, I started working my way back through the historical database from the laboratory.

It was a lot of work. A lot. Luckily my PI at the time was okay with it and seemed to think I was being productive. Some of the new stuff that I was doing with our data stream ended up being included by default in most of our publications thereafter. Some of it ended up in its own publication, albeit some 12 years after I had completed the initial data mining. (This latter paper has barely ever been cited but I still think the result is super cool.) The data mining of files from experiments that were run before I entered the laboratory required a second bit of work, as you might readily imagine. I had to parse back through the lab books to find out which subject numbers belonged together as cohorts or experiments. I had to separate training data from baseline / maintenance studies, from experimental manipulations of acute or longitudinal variety. And examine these new data extractions in the context of the actual experiment. None of this was annotated in the files themselves. There wasn’t really a way to even do it beyond 8 character file names. But even if it had been slightly better curated, I’m just not seeing how it would be useful without the lab books and probably some access to the research team’s memory.

Snapping forward to me as a PI, we have somewhat similar situation in my lab. We have a behavioral assay or two run by proprietary commercial software that generate data files that could, in theory, be mined by anyone that was interested* in some aspect of the behavior that struck their fancy. It would still take a lot of work and at least some access to the superordinate knowledge about the studies a given subject/date stamped file related to. I am happy for trainees in my lab to play with the data files, present and past. I’m happy for them to even replace analysis and reporting strategies that I have developed with their own, so long as they can translate this to other people in the lab. I.e., I am distinctly unkeen on the analysis of data being locked up in the proprietary code or software on a single trainee’s laptop. If they want to do that, fine, but we are going to belt-and-suspenders it. There is much value in keeping a set of data analysis structures more or less consistent over time. Sometimes the most rudimentary output from a single data file (say, how many pellets that rat earned) is all that we need to know, but we need to know that value has been used consistently across years of my work.

I have at least two interests when it comes to data curation in my lab. I need some consistency and I need to be able to understand as the PI what I am looking at. I need to be able to go back to some half-remembered experiment and quickly whip up a preliminary data or slide figure. This leans towards more orthodoxy of analysis. Towards orthodoxy of data structures and formats. Towards orthodoxy in the graphs, for pete’s sake. My attempts to manage this into reality has mixed results, I will note. At the level of an individual staffer, satisfying some data curation goal of the PI (or anyone else, really) can seem like make-work. And it is definitely work to the ends of someone else, I just happen to be the PI and am more equal that anyone else. But it is work. And this means that short cuts are taken. Often. And then it is down to the effort of someone to bring things back up to curation standard. Sure it may seem to be “just as easy” for the person to do it the way I want it, but whaddayaknow, they don’t always see it that way. Or are rushed. Or mean to get to that at the end of the study but then forget. Tomorrow. When it is really needed.

I get this. It is a simple human reality.

In my lab, I am the boss. I get to tell staff members what to do and if they won’t do it, eventually, I can fire them. Their personal efforts (and mine for that matter) are supposed to be directed towards the lab good, first, and the institutional good second. The NIH good is in there somewhere but we all know that since a grant is not a contract, this is a very undefined concept.

There is very little that suggests that the effort of my laboratory staff has to be devoted to the good of some other person who wants access to our data in a way that is useful to them. In fact, I am pretty sure in the extreme case that if I paid a tech or trainee from my grant to work substantial amounts of time on a data analysis/curation project demanded of us by a private for-profit company solely for their own ends, this would violate the rules. There would probably be a technical violation if we did the same for a grant project funded to another researcher if the work had nothing whatever to do with the funded aims in my own lab that were paying the staff member’s salary.

Data curation for others’ ends costs. It costs time and that means that it costs money. It is not trivial. Even setting up your data stream within lab so that it could possibly be easier to share with external data miners costs. And the costs apply to all of the data collected, not just that that eventually, one day is requested of you and ends up in a paper.

__

*as it happens we just fielded a request but this person asked us to collaborate, rightfully so.

Trophy collaborations

July 5, 2018

Jason Rasgon noted a phenomenon where one is asked to collaborate on a grant proposal but is jettisoned after funding of the award:

I’m sure there are cases where both parties amicably terminate the collaboration but the interesting case is where the PI or PD sheds another investigator without their assent.

Is this common? I can’t remember hearing many cases of this. It has happened to me in a fairly minor way once but then again I have not done a whole lot of subs on other people’s grants.

A new Op/Ed in PNAS takes up the reproducibility crisis.

A. D Redish, E. Kummerfeld, R. L. Morris, A. Love (2018) “Opinion: Reproducibility failures are essential to scientific inquiry” PNAS 115(20):5042-5046. [Journal Site]

Takeaway quote from the Abstract

Most importantly, these proposed policy changes ignore a core feature of the process of scientific inquiry that occurs after reproducibility failures: the integration of conflicting observations and ideas into a coherent theory.

As you will see, they had me at:

In many of these cases, what have been called “failures to replicate” are actually failures to generalize across what researchers hoped were inconsequential changes in background assumptions or experimental conditions

(Oh, wait, they cited me! AHAHAA, of course I like this thing!)

Seriously though, this is good stuff. Go read. Bookmark to forward to anyone who starts in on how there is a reproducibility “crisis”.

On the May 1, 2018 the NIH issued NOT-OD-18-172 to clarify that:

NIH seeks to remind the extramural community that prior approval is required anytime there is a change in status of the PD/PI or other senior/key personnel where that change will impact his/her ability to carry out the approved research at the location of, and on behalf of, the recipient institution. In particular, changes in status of the PI or other senior/key personnel requiring prior approval would include restrictions that the institution imposes on such individuals after the time of award, including but not limited to any restrictions on access to the institution or to the institution’s resources, or changes in their (employment or leave) status at the institution. These changes may impact the ability of the PD/PI or other senior/key personnel to effectively contribute to the project as described in the application; therefore, NIH prior approval is necessary to ensure that the changes are acceptable.

Hard on the heels of the news breaking about long term and very well-funded NIH grant Principal Investigators Thomas Jessel and Inder Verma being suspended from duties at Columbia University and The Salk Institute for Biological Studies, respectively, one cannot help but draw the obvious conclusion.

I don’t know what prompted this Notice but I welcome it.

Now, I realize that many of us would prefer to see some harsher stuff here. Changing the PI of a grant still keeps the sweet sweet indirects flowing into the University or Institute. So there is really no punishment when an applicant institution is proven to have looked the other way for years (decades) when their well-funded PIs are accused repeatedly of sexual harassment, gender-based discrimination, retaliation on whistleblowers and the like.

But this Notice is still welcome. It indicates that perhaps someone is actually paying a tiny little bit of attention now in this post-Weinstein era.

NIH encourages pre-prints

February 13, 2018

In March of 2017 the NIH issued a notice on Reporting Preprints and Other Interim Research Products (NOT-OD-17-050): “The NIH encourages investigators to use interim research products, such as preprints, to speed the dissemination and enhance the rigor of their work.“.

The key bits:

Interim Research Products are complete, public research products that are not final.

A common form is the preprint, which is a complete and public draft of a scientific document. Preprints are typically unreviewed manuscripts written in the style of a peer-reviewed journal article. Scientists issue preprints to speed dissemination, establish priority, obtain feedback, and offset publication bias.

Another common type of interim product is a preregistered protocol, where a scientist publicly declares key elements of their research protocol in advance. Preregistration can help scientists enhance the rigor of their work.

I am still not happy about the reason this happened (i.e., Glam hounds trying to assert scientific priority in the face of the Glam Chase disaster they themselves created) but this is now totally beside the point.

The NIH policy (see OpenMike blog entry for more) has several implications for grant seekers and grant holders which are what form the critical information for your consideration, Dear Reader.

I will limit myself here to materials that are related to standard paper publishing. There are also implications for materials that would never be published (computer code?) but that is beyond the scope for today’s discussion.

At this point I will direct you to bioRxiv and PsyRxiv if you are unfamiliar with some of the more popular approaches for pre-print publication of research manuscripts.

The advantages to depositing your manuscripts in a pre-print form are all about priority and productivity, in my totally not humble opinion. The former is why the Glamour folks are all a-lather but priority and scooping affect all of us a little differently. As most of you know, scooping and priority is not a huge part of my professional life but all things equal, it’s better to get your priority on record. In some areas of science it is career making/breaking and grant getting/rejecting to establish scientific priority. So if this is a thing for your life, this new policy allows and encourages you to take advantage.

I’m more focused on productivity. First, this is an advantage for trainees. We’ve discussed the tendency of new scientists to list manuscripts “in preparation” on their CV or Biosketch (for fellowship applications, say, despite it being technically illegal). This designation is hard to evaluate. A nearing-defense grad student who has three “in prep” manuscripts listed on the CV can appear to be bullshitting you. I always caution people that if they list such things they had better be prepared to send a prospective post-doc supervisor a mostly-complete draft. Well, now the pre-print allows anyone to post “in preparation” drafts so that anyone can verify the status. Very helpful for graduate students who have a short timeline versus the all too typical cycle of submission/rejection/resubmission/revision, etc. More importantly, the NIH previously frowned on listing “in preparation” or “in review” items on the Biosketch. This was never going to result in an application being returned unreviewed but it could sour the reviewers. And of course any rule followers out there would simply not list any such items, even if there was a minor revision being considered. With pre-print deposition and the ability to list on a NIH biosketch and cite in the Research Plan there is no longer any vaporware type of situation. The reviewer can look at the pre-print and judge the science for herself.

This applies to junior PIs as well. Most likely, junior PIs will have fewer publications, particularly from their brand new startup labs. The ability of the PI to generate data from her new independent lab can be a key issue in grant review. As with the trainee, the cycle of manuscript review and acceptance is lengthy compared with the typical tenure clock. And of course many junior PIs are trying to balance JIF/Glam against this evidence of independent productivity. So pre-print deposition helps here.

A very similar situation can apply to us not-so-junior PIs who are proposing research in a new direction. Sure, there is room for preliminary data in a grant application but the ability to submit data in manuscript format to the bioRxiv or some such is unlimited! Awesome, right?

Thought of the Day

January 4, 2017

While I think generosity on the part of more senior scientists is a good thing, and should be encouraged, making this an obligation is flawed. How do you know what that person’s obligations are?

I post this in case any PI types out there don’t know this is a thing. If you can pick up a check or pay more than your share, hey great. Good for you.

But nobody should expect it of you.

Finishing projects

December 30, 2016

If you are paid by the taxpayers, or generous private philanthropists, of your country to do science, you owe them a product. An attempt to generate knowledge. This is one of the things that orients much of my professional behavior, as I think I make clear on this blog.

If you haven’t published your scientific work, it doesn’t exist. This is perhaps an excessive way to put it but I do think you should try to publish the work you accomplish with other people’s money.

Much of my irritation with the publication game, prestige chasing, delusions of complete stories, priority / scooping fears and competition for scarce funding resources can be traced back to these two orienting principles of mine.

My irritation with such things does not, however, keep them from influencing my career. It does not save me from being pressured not to give the funders their due.

It is not unusual for my lab, and I suspect many labs, to have thrown a fair amount of effort and resources into a set of investigations and to realize a lot more will be required to publish. “Required”, I should say because the threshold for publication is highly variable.

Do I throw the additional resources into an effort to save what is half or three-quarters of a paper? To make the project to date publishable? I mean, we already know the answer and it is less than earth shaking. It was a good thing to look into, of course. Years ago a study section of my peers told us so to the tune of a very low single digit percentile on a grant application. But now I know the answer and it probably doesn’t support a lot of follow-up work.

Our interests in the lab have moved along on several different directions. We have new funding and, always, always, future funding to pursue. Returning to the past is just a drag on the future, right?

I sometimes feel that nobody other than me is so stupid as to remember that I owe something. I was funded by other people’s money to follow a set of scientific inquiries into possible health implications of several things. I feel as though I should figure out how to publish the main thing(s) we learned. Even if that requires some additional studies be run to make something that I feel is already answered into something “publishable”.

The very first rule of PI/mentorship is get your trainees first author publications.

This is the thing of biggest lasting career impact that you can determine almost with absolute control.

Yes, things happen but if you are not getting the vast majority of your trainees first author pubs you are screwing up as a mentor.

So. 2017 is about to start. Do you have a publication plan for all of your postdocs and later-stage graduate students?

Obviously I am in favor of active management of trainees’ publishing plans. I assume some favor a more hands-off approach?

“Let the postdoc figure it out” has an appeal. Makes them earn those pubs and sets them up for later hard times.

The problem is, if they fail to get a publication, or enough, their career takes a bad hit. So ability to grunt it out isn’t ever used.

Go!