Manuscript review and the alleged reproducibility crisis

If you believe there is a “replication and reproducibility crisis” in science, you have limited proposal options.

Multiple independent replications of each phenomenon in a paper. Not n-up, but triplicate or more full power repeats.

Are you demanding this? Are you doing this in your own research?

Or, we create some way to encourage and enhance direct replications and simple tests of generalization from other labs or in subsequent papers.

I favor this one.

As it happens,I have had multiple incidences in the past few years which address this. As both an author and as a reviewer.

Editors and reviewers for decidedly modest JIF journals are overtly and explicitly saying replications and simple tests of generalization of a finding should not be published.

I can’t stress this enough. We’re not talking “this isn’t cool enough for this 2.5 JIF journal”. These are opinions that such things do not merit publication at all.

Findings that fail to replicate a prior finding (that is actually poorly supported) *simultaneously* take heat for not getting the same result.

Direct replication is too incremental and refutation / modification is too…..doubtful?

As my longer term Readers know, I tend to think this is just the way science works. If you keep at it your manuscript will find a home eventually. It is a PIA but it is not damning of the entire enterprise.

But if there is any validity to the reproducibility claims and you keep braying on about it…I want to know a lot more about how your reviewing behavior matches your fine talk.

I’m also looking forward to NIH grant review in about 3-4 years. We should be expecting the experienced PI to have some specific and concrete examples about their rigor.

Even if it is “this preprint shows our replication attempts, even if idiotic reviewers prevented them from being published”.