Addressing the File Drawer Problem of (Non)Replication in Science

This is absolutely BRILLIANT!

PsychFileDrawer.org is:

a tool designed to address the File Drawer Problem as it pertains to psychological research: the distortion in the scientific literature that results from the failure to publish non-replications. Most journals (especially high impact journals that specialize in publishing surprising findings that have low prior odds of being correct) are rarely willing to publish even carefully conducted non-replications that question the validity of a finding that they have published. Often the only people who learn about non-replications are those who happen to be “plugged in” to social networks that circulate this information in a fragmentary and inefficient way. Even textbook authors are rarely well informed about the replicability of the results that they report on, and may often rely upon results that are known to be dubious by those working in the area.

What a great idea. One of the reasons I recently held out as a justification for the LPU approach to publishing is the hoarding of not-enough-for-pub data out there that might save someone else a whole hell of a lot of time. Well, chasing after a supposed published finding as your control or launching point for new studies can land you in one of those little potholes. Wouldn’t it be nice to see a half dozen (or more) attempts to replicate an effect to (at the very least) tell you which are the key conditions and which can be manipulated for your purposes?

Other fields should try something like this.
__
Disclaimer: I’m professionally acquainted with one or more of the people apparently involved in this effort.

Regressive taxation schemes and J&D psychology sneering aside, this proves we have the best job imaginable

You may have noticed lottery mania has reached a bit of a peak lately what with the over $640 Million prize anticipated in the 42-state Mega Millions drawing.

Of course if you win tonight and take the cash payout on the $640 million, you’d get $462 million, according to the lottery. Or you could opt to receive annual payments of $24 million a year for 26 years (before taxes).

The Lefties will inform you of the great evils of lottery as a replacement for a more progressive tax scheme. Point out that it is mostly the unwashed, poor and working-person crowd that throws down the most support for lottery. Bah, you say, “buzzkills!”. Well, perhaps……

Likewise, the Judgment and Decision experimental psychologists will blather on about how lottery playing is a sucker’s game. And yes indeed it is. Because the stats don’t work out as optimal in gaming the end stage of large amounts of play. Of course, people who actually know something about judgment will tell them that when the entry price is low (like a buck a ticket) and the payoff is not just the chance of a fabulous, lifechanging amount but also the pleasant fantasy part in anticipating what you would do with the money….

genomicrepairman indulged in a little bit of fantasy himself and I do admit that is a damn fine idea.

I thought about this long and hard for about three seconds on my drive in today after scanning GR’s post at a stoplight.

I conclude this.

$24 million per year for a quarter of a century….hmmmm. Ok, let’s just ballpark the taxes as half and round down to $10M. That’s some chazange. And I’m sure a spouse would have something to say about the prize so let’s drop that to a cool $5M per year. And let’s suppose some of you are out there in institutions with 100% overhead for charitable giving. That leave’s you with $2.5M to pour into your laboratory’s direct-cost fund each and every year!

Imagine that. No more worries about funding your lab. No need for a grant at all. Pure, endowed chair and endowed slush fund. Schweet.

All you have to do is make it to tenure and the University would have a damn hard time getting rid of you for just about anything. Even if you fail to publish for a year or two while you are working on some harebrained idea you had.

Are you telling me you would quit your position to tour the earth? Hell, if you have a half decent lab manager and a post doc or two you could still tour the earth, surfacing every couple of weeks in an obscure Internet cafe in Barbaloot to check in!

There is no way in hell I’d quit my job if I won this lotto. Take the financial headaches away and it turns from being a damn good job to the best thing to do in the entire world, if you ask me….you feel any differently?

__
ps. It would have to be like this.

We said we would tell no one. We would hire an attorney and anonymously collect our winnings under the cover of a trust. We would not take any publicity photos or announce our newfound wealth with conspicuous consumption. We might get a pool because we all love to swim and it’s good exercise. But generally no one would guess that my husband and I had struck it rich.

There are just too many examples of former lottery winners who end up losing their millions because they overspent and overindulged themselves.

sekrit trust, right. otherwise the postdocs wouldn’t work hard enough….

On technology, a quick poll

At what age did you get your first cell phone and what excuse reason drove your decision? (I ask due to a tangential line of thought emerging from comment from a LaTex fan decrying the technological conservatism of the Academy.)

I was 31 and there was wedding planning involved.

Elsevier claims publishers prevent….protect…wait, what now?

This is hilarious.

In addition, from an editorial perspective, copyright helps to prevent elements such as plagiarism, multiple submission and fraud in journal articles, and whilst is does not actually detect these elements, it acts as a protective measure to uphold the quality of journals.

What the hell did this say? They can’t detect shenanigans (score another to post-pub processes) but they…huh? What? Reluctantly post retractions? Send out a cease and desist letter from the same bunker that hit @FakeElsevier?

This doesn't sound "mean" to me

Some commenter at Rock Talk complained about a recent grant review:

I just received the most terrible of reviews, where the reviewer was not only biased but highly inflammatory, prejudicial and aggressive. I must say I was totally taken aback. When you say things like “…terribly convoluted approach”,…”PI has clearly no clue…” how something works, trashes my published work by saying these pubs “are a gross exaggeration”….the list goes on and on. Even as a relatively senior investigator, I was very shocked by the mean-spirited nature of the comments. I cannot imagine how it would destroy a new investigator.

I am having trouble seeing it. I mean sure “no clue” is directed at the applicant rather than the application, but it’s pretty tame stuff. If a reviewer thinks your papers exaggerate? Presumably in wild speculative interpretation that runs beyond your data? Seems okay and even obligatory to express this. The “terribly convoluted approach” comment is a pretty inoffensive way to get to the heart of this common failing of grants as well…I’m not seeing how you could put it more “nicely”.

Actually, college profs work hard for poor compensation

Some dim bulb named David Levy has trotted out a fairly impressive amount of ignorance about college and University professors in the US. Honestly, I’m failing to see one substantive point that matches reality for the vast majority of professors (and adjunct instructors) today. This guy is sadly out of touch or cherry picking from a handful of anecdotes…..or just making it up. Maybe this is one of those works of performance art? A Kylian “not intended to e a factual statement”?

Allow Dr Zen to retort.

K22

The NIH K22 mechanism has been around for some time. When I first discovered it I was amazed that the NIH had a genuine, Borroughs-Welcome style transition mech. This predated the K99/R00.

Trouble was, most (all?) ICs that used it were like NHLBI and reserve it for intramural postdocs.

What a crock.

Query for the day

Why are so many parents in such denial about their kids’ behavior? How can they have forgotten what they were like as kids and how their friends and neighbors behaved?

That’s the kid space. It hasn’t changed for the better. Your children occupy it. They are not magically angels just because they are yours.

So somewhere in your mind you should recognize it is possible that your kid would “do that”.

Francis Collins assures Congress that NCATS won't draw from basic research

via the Nature News Blog we learn that NIH Director Collins has been called upon by Congress to explain NCATS. This is the acronym for his pet project a Center dedicated to “Translational Science” that required axing the venerable National Center for Research Resources.

Collins noted that NIH’s support for basic research has held steady at about 54% of the agency’s budget for decades. “I do not expect that percentage to change,” he said. He added repeatedly that all but 2% of the $575 million funding the translational medicine center this year comes from preexisting NIH programs, and is not “new” money.

Some legislators, understanding quite clearly that money is fungible were keen to press Collins:

Representative Cynthia Lummis, Republican of Wyoming, interrupted Collins to insist that he explain how the $64 million increase proposed for NCATS in 2013 can’t be seen as being largely funded by a cut to the Institutional Development Award (IDEA) program. The NIH in 2013 has proposed cutting $50.5 million from the program, which funds biomedical investigators, trainees and infrastructure in 23 largely rural states that have historically experienced low application success rates for NIH grants.

“I would not want you to see a direction connection between…the IDEA program and NCATS. Those are not the same dollars that just got moved from one box to another,” Collins responded.

“Dollars are dollars,” Lummis replied.

Exactly. And similarly there are plenty of imaginable grants that would be “translational science” that Collins will get to score in the “basic” category as well. Another CongressCritter argued with Collins that a prior boost to the IDEA program was intended to be permanent, something Collins disputed. Yes, keeping track of this slippery customer down the line will be pretty hard for our intrepid Congressional heroes.

There was another bit of testimony that drew my eye because it speaks to the potential upside of NCATS rather than whinging (ahem) about the costs to other programs. NCATS is supposed to somehow do better than the pharma industry. Ok, fine, but it sort of presumes the pharma industry is full of morons*. I’ve seen this before from academics under various guises of “Rational Drug Design” and the like. I am, shall we say, skeptical. In this particular bit of testimony on the “we’re smarter than they are, nyah, nyah” defense for NCATS a BigPharma type observed that FC is full of stuff and nonsense:

That view was challenged later in the hearing by Roy Vagelos, the former CEO of Merck, who said that the pharmaceutical industry spends about $50 billion annually, or roughly 100 times the NCATS budget, without solving the problems, like inadequate toxicology, that cause so many failures in drug development . “Does anyone in the audience believe that there is something that NCATS is going to do that the industry thinks is critical and that they are not doing? That is incredible to think that. If you believe that you believe in fairies.”

Vagelos added that, with success rates for applicants for NIH grants at historic lows, “We would be doing a lot more good for getting important new drugs on the market,” by funding more young investigators.

Word, PharmaDude, word.

__
*unless it addresses things that the profit industry is not really capable of grappling with such as their penchant for huge payoff, block buster, serves everyone type of drugs.

Authors fail to illuminate the LPU issue

We most recently took up the issue of the Least Publishable Unit of science in the wake of a discussion about first authorships (although I’ve been talking about it on blog for some time). In that context, the benefit of having more, rather than fewer, papers emerging from a given laboratory group is that individual trainees have more chance of getting a first-author slot. Or they get more of them. This is highly important in a world where the first-author publications on the CV loom so large. Huge in fact.

I’ve also alluded to the fact that LPU tendencies are a benefit to the conduct of science (as a group enterprise) because it allows the faster communication of results, the inclusion of more methodological detail (critical for replication and extension) and potentially the inclusion of more negative outcomes (which saves the group time).

I have also staked my claim that in an era when most of us find, sort and organize literature with search engine tools from our desktop computers, the “costs” of the LPU approach are minimal.

The recent APS Observer reprinted a column in the NYT that I’d originally missed entitled “The Perils of ‘Bite sized’ Science” (MARCO BERTAMINI and MARCUS R. MUNAFÒ; Published: January 28, 2012 ). Woot! No offense, commentariat, but you’ve done a dismal job so far of making an argument for why the LPU approach is so bad or detrimental to the conduct of science, particularly in response to my reasons. So I was really stoked to see this, in hopes of gaining some insight. I was sadly disappointed.

Summary Statement Paranoia

An interesting viewpoint popped up on writedit’s blog over the past couple of days. A commenter who is apparently not yet in an Assistant Professor position has managed to get a nice little grant score:

My RO1 is scored 23 with a percentile is 10. I am now applying for an AP position.

Sweet. Nice accomplishment. Where’s the problem?

An University that I just visited aksed me to email them the summary statement. I was wondering if this is appropriated?

Ummmm, well, sure. I mean, particularly if this applicant has informed the search committee / Chair / Dean that s/he has this particular score in hand. Let us be clear. Many, many hiring departments are trying to game out the potential each new hire has for funding. Funding in the short term and funding in the long term. Funding is tight all over, including internal pilot funds, and all things equal the Department wants to hire someone who can get and sustain major funding. Rapidly.

Because that is the lynchpin to scholarly research (again, for many research-oriented job categories), to productivity, to tenure and ultimately to burnishing the Department’s reputation.

In some places we are even hearing rumour that search committees are not really considering anyone who doesn’t already come with some sort of research funding. This can be via informal or formal (Dean diktat) rules…or merely via competition within the applicant pool.

Consequently, any evidence of a fundable score on a NIH grant is pretty meaningful. Meaningful to your chances before you make the short list and even more meaningful once they’ve brought you out to interview.

Correspondingly I think the applicant would be EAGER to show off a score like this and send off the summary statement post haste.

Apparently opinions vary:

My RO1 is currently pending and the council meeting will be in May. If another university, not my current institution, contacts my program officer, it will mean that I am leaving from my current institution. The score of a RO1 is based on the environment of applicant’s current institution. Will leaving this environment be considered a negative factor by the council meeting and the program officer, and thus influence their decision on whether they would like to fund my RO1 or not?

additional crosstalk went off the rails. writedit: They should only need the first page in that case.
more writedit: The request for the summary statement is not appropriate, especially so early in the application/interview stage.
the original commenter ate this up:

I probably will send them the first page of the SS, but will mask the confidential information, such as the PO contact information and my application ID. This might offend the prospective univeristy, to avoid this, I may call them first to let them know my concern.

and some other commenter did as well:

The bigger concern is perhaps that there may specific criticisms regarding you, your team, and the institution. There may be comments in SS that highlight any weaknesses that are really not the business of the perspective institution.

This is nuts. Absolute nuts. First, the PO is going to be happy that the applicant has secured a tenure track appointment. This is a good thing. It is doubtful this is going to put the PO off funding the application. Unless there is something in the way of highly unusual circumstances like a unique resource or some spendy equipment that cannot be duplicated at the new place.

Second, the prospective University is not going to contact the PO….what would they be asking anyway? Whether some other University’s grant application will be funded? The PO will tell them to butt out.

Third, criticisms in a tenth percentile summary statement are going to put the hiring University off? If they are this stupid, this person doesn’t want to work there, if you ask me. Seriously though, what are they going to read? “This Investigator is well qualified and promising”, that’s about it.

Finally, all this “masking” the grant number and “only the first page” stuff is seriously silly. All it is going to do is make the prospective University wonder what sort of paranoid nutjob they have on the line and whether they’ve made a serious mistake in trying to hire this wackaloon.

NIH Advice for Noobs on the Twitts today

Your roving reporter, @doc_becca (of the Fumbling Towards Tenure blog) has a few observations rolling under the #NIHsekrits hashtag on the Twitts today. Check it out.

Perfection

The notion that there is some perfect pedigree, some perfect CV that most applicants to the NIH (or other funding body, I assume) possess is untrue.

Oh, it can be comforting, I realize. To think that if you only had picked the right doctoral or post doc lab, if only you had been able to move across the country, if only that damn PI had written you a better letter and gotten you that job at HighFalutinU….

If only.

But for those slings and arrows of outrageous fortune, you too would get every application funded. Nary a triage and seldom a revision.

Hahahahahhahahahahaaaa!!!!

Get real.

Many, many funded PI’s of my acquaintance have holes in their CVs that you can drive the ParanoidApplicantTrain through. The wrong doctoral Uni/program. Dismal productivity in a crucial stop. No post doc. Too many postdoc years. A less-than appointment. Seeming lack of independence. A mid career research drought. Low IF pubs. A scientific diversion. Too narrow a scientific focus. Too diffuse.

The thing is…the extramural, NIH-funded community is diverse on this axis. It permits a lot of room for Investigators who differ from the “ideal”. To the degree that the “ideal” is more of a fantasy of the unsuccessful applicant than it is a reality.

My charge to you is this: Ignore your seeming deficits.

Ignore the inner voice telling you that easy street was back there on the path not taken. Or the path that was barred to you.

Focus instead on crafting the story of you, as a scientist and investigator. How did your experiences make you the independent scientist that you are?

Remember that you are not talking to your detractors but rather to your advocates.

More recipes for the busy scientist

Recipe:

Swing by Trader Joes, pick up

-Prepared herb pizza dough

-mushrooms

-pesto

-mozzarella prosciutto log thingy

-arugula

Slice a couple of mozzarella thingies whilst lightly cooking the mushrooms. Wilt arugula.

Quarter the dough, spray some oil on a flat sheet. Smooch out the dough, getting it oily on both sides.

Aliquot arugula, mushroom and mozzarella / prosciutto on the dough, splash on some pesto. Wrap it over and pinch the edges together. (I didn’t use anywhere near enough arugula on this one. And a little garlic would have been nice.)

Bake at 400 for 10-12 minutes.

As FizzyPoof says…..

Eatte itte!!!!

Alcohol research fraudster Michael W. Miller

Per ORI, one Michael W Miller, most recently Chair of Neuroscience and Physiology at SUNY Upstate Medical University, is a data faker.

ORI finds that the Respondent engaged in research misconduct by falsifying and/or fabricating data that were included in grant applications R01 AA07568-18, R01 AA07568-18A1, R01 AA006916-25, and P50 AA017823-01 and in the following:

The “following” included some paper retractions detailed over at the RetractionWatch blog.

One of the interesting things is that this guy published in decidedly normal journals. There was something in the ORI finding about a PNAS submission, but that seemed to be the high IF watermark for Miller. I make special note of this since I am one of those fond of pointing out the positive correlation between journal IF and retractions.

You will be unsurprised that my attention is drawn in this case to the grant support. That year -18 grant renewal application mentioned? It got converted into an R37 MERIT (10 yrs of non competing renewals instead of the usual 5) in the A1 version. Which means it was scored very highly, from what I deduce about the R37 process. The P50 is, of course, a Center.

Big monetary commitment for NIAAA and very prestigious for Professor Miller.

Boo, hiss, fraudster bad guy….

Except think about those folks who didn’t get something because if this guy. The Chair position was an external hire. The P50 took the place of another one- and it isn’t just the PI/PD. Each competing Center that didn’t get funded probably also had a handful of Component PIs. Who put in a lot of hard work and had a lot of great science. The R37? Well it probably counts at least double because of the 10 year interval. And of course some other worthy mid-career NIAAA scientist didn’t receive this honor for her work.

I’m irritated on behalf of anyone who applied to NIAAA for grant support and didn’t get the award during the interval they were supporting this Miller fraudster.

Final note. He was supposedly ratted out by someone in his lab. Since the offense seemed to be making up bar graphs rather than the all-too-typical duplicated gel/blot/image, there really would have been no other way to nail him. So the reviewers really can’t be blamed for missing anything.