Being as wrong as can be on the so-called replication crisis of science

I am no fan of the hysterical hand wringing about some alleged “crisis” of science whereby the small minded and Glam-blinded insist that most science is not replicable.

Oh, don’t get me wrong. I think replication of a prior result is the only way we really know what is most likely to be what. I am a huge fan of the incremental advance of knowledge built on prior work.

The thing is, I believe that this occurs down in the trenches where real science is conducted.

Most of the specific complaining that I hear about failures to replicate studies is focused on 1) Pharma companies trying to cherry pick intellectual property off the latest Science, Nature or Cell paper and 2) experimental psychology stuff that is super truthy.

With regard to the former, cry me a river. Publication in the highest echelons of journals, publication of a “first” discovery/demonstration of some phenomenon is, by design, very likely not easily replicated. It is likely to be a false alarm (and therefore wrong) and it is likely to be much less generalizable than hoped (and therefore not “wrong” but definitely not of use to Pharma vultures). I am not bothered by Pharma execs who wish that public funded labs would do more advancing of intellectual property and serve it up to them part way down the traditional company pipeline. Screw them.

Psych studies. Aaah, yes. They have a strong tradition of replication to rely upon. Perhaps they have fallen by the wayside in recent decades? Become seduced to the dark side? No matter. Let us return to our past, eh? Where papers in the most revered Experimental Psychology journals required several replications within a single paper. Each “Experiment” constituting a minor tweak on the other ones. Each paper firmly grounded in the extant literature with no excuses for shitty scholarship and ignoring inconvenient papers. If there is a problem in Psych, there is no excuse because they have an older tradition. Or possibly some of the lesser Experimental Psychology sects (like Cognitive and Social) need to talk to the Second Sect (aka Behaviorism).

In either of these situations, we must admit that replication is hard. It may take some work. It may take some experimental tweaking. Heck, you might spend years trying to figure out what is replicable / generalizable, what relies upon very ….specific experimental conditions and what is likely to have been a false alarm. And let us admit that in the competitive arena of academic science, we are often more motivated by productivity than we are solving some esoteric problem that is nagging the back of our minds. So we give up.

So yeah, sometimes practicalities (like grant money. You didn’t seriously think I’d write a post without mentioning that, did you?) prevent a thorough run at a replication. One try simply isn’t enough. And that is not a GoodThing, even if it is current reality. I get this.

But….

Some guy has written a screed against the replication fervor that is actually against replication itself. It is breathtaking.

All you need to hook your attention is conveniently placed as a bullet point pre-amble:

· Recent hand-wringing over failed replications in social psychology is largely pointless, because unsuccessful experiments have no meaningful scientific value.
· Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way. Unless direct replications are conducted by flawless experimenters, nothing interesting can be learned from them.
· Three standard rejoinders to this critique are considered and rejected. Despite claims to the contrary, failed replications do not provide meaningful information if they closely follow original methodology; they do not necessarily identify effects that may be too small or flimsy to be worth studying; and they cannot contribute to a cumulative understanding of scientific phenomena.
· Replication efforts appear to reflect strong prior expectations that published findings are not reliable, and as such, do not constitute scientific output.

· The field of social psychology can be improved, but not by the publication of negative findings. Experimenters should be encouraged to restrict their “degrees of freedom,” for example, by specifying designs in advance.

· Whether they mean to or not, authors and editors of failed replications are publicly impugning the scientific integrity of their colleagues. Targets of failed replications are justifiably upset, particularly given the inadequate basis for replicators’ extraordinary claims.

Seriously, go read this dog.

This part seals it for me.

So we should take note when the targets of replication efforts complain about how they are being treated. These are people who have thrived in a profession that alternates between quiet rejection and blistering criticism, and who have held up admirably under the weight of earlier scientific challenges. They are not crybabies. What they are is justifiably upset at having their integrity questioned.

This is just so dang wrong. Trying to replicate another paper’s effects is a compliment! Failing to do so is not an attack on the authors’ “integrity”. It is how science advances. And, I dunno, maybe this guy is revealing something about how he thinks about other scientists? If so, it is totally foreign to me. I left behind the stupid game of who is “brilliant” and who is “stupid” long ago. You know, when I was leaving my adolescent arrogance (of which I had plenty) behind. Particularly in the experimental sciences, what matters is designing good studies, generating data, interpreting data and communicating that finding as best one can. One will stumble during this process…if it were easy it wouldn’t be science. We are wrong on a near-weekly basis. Given this day to day reality, we’re going to be spectacularly wrong on the scale of an entire paper every once in awhile.

This is no knock on someone’s “integrity”.

Trying to prevent* anyone from replicating your work, however, IS a knock on integrity.

On the scientific integrity of that person who does not wish anyone to try to replicate his or her work, that is.

__
*whether this be by blocking publication via reviewer or editorial power/influence, torpedoing a grant proposal, interfering with hiring and promotion or by squelching intrepid grad students and postdoctoral trainees in your own lab who can’t replicate “The Effect”.