I really don't give a flying fig about your interpretation of your data

January 10, 2012

SciCurious has a post up today, rambling on about some apparent bugaboo of the trainee set. There’s a survey and everything so go read it.

What I take away from this is that there is some aspect within the culture of science in which there are some bragging rights to be obtained on the basis of how many papers you read. “Read“.

This is asinine.

Some commenter named ‘brian’ irritated me with:

I disagree with these definitions of reading. An abstract and figures are in no way reading a paper. To truly grasp what authors are trying to say you need to have a working knowledge of the entire article, intros and discussions are critical since bad assumptions and wrong interpretations are the most likely errors in any paper. Abstract and figures and a quick skim of the discussion are a cop out way of upping your read list.

This comment seems to personify that subject which is SciCurious original motivation. People in science who are really, really focused on how many articles they read and the nature of the “reading” of such articles.

Bollocks.

Science is about apprehending what has and has not been demonstrated on a particular subject. About understanding what has been supported by evidence, what may be provisionally inferred and what has been provisionally (or repeatedly) rejected. As a practicing scientist, this understanding permits us to design better or more interesting experiments, make better or more interesting interpretations of our data and set better or more interesting provisional inferences. There is no score card for “how many papers I have read”. There is only a scorecard for how great your science chops are.

It therefore follows that the nature of reading a scientific paper may vary tremendously in the service of the real goal. The nature of this reading may not be the same across different scientists (who are interested in different things) or even within scientist across time. Some details in a paper may be irrelevant to one person, but highly pertinent to another. Details of methodology (who gives a crap what kind of rat they used…a rat’s a rat….until I happen to be interested in a strain difference, that is) or of outcome (quarter-log, half-log shift in the dose-response function, who cares? we’ll just up the starting point for the clinical, titrated dose.) and especially of the interpretation (dude, I don’t care one bit whether this drug is likely to be abused recreationally, it’s just a good probe for the endogenous system…) will vary.

For the trainees, I sympathize.

I understand, I think, how you come to this misunderstanding of paper count as a measure in and of itself of scientific acumen. It is because over the course of a scientist’s lifetime, she reads a hella lot of papers and draws together a hella lot of stories. So for the noob(ier) scientists, the scope of understanding of the most impressive and vigorous scientists seems a little daunting. Intimidating. Because you are looking at what seems an unbounded ability to reel off citations of relevant papers in the service of some point the Big Swinging type is making. But this is natural accumulation. It builds up over time.

And here’s a little hint. The brilliant types with seemingly unbounded ability to reel off citations don’t “read” all the literature in the way brian, above, would have it either. They read in a variable fashion.

Sometimes it is just a glance at Figure 2. Sometimes it is the deepest of deep readings and pondering the paper over several days that is required. Sometimes it is a series of re-reads over months or years. Always, in service of the real goal which is to understand how the data presented in a paper fit into a scientific story that is of interest.

Never in service of bragging to one’s peers about how many papers one has “read’ this month.

I mean Jesus, do you even listen to yourselves? Doesn’t that sound insane?

There is another reason for the quick glance-at-the-Figures level of reading which trainees find incomprehensible because it is so damn insulting. I really don’t care what you think about your data. I care what I think about your data.

This is a horrible realization for trainees that spend many, many long hours crafting their rationale and interpretation in the Discussion section. Who have also spent many long hours arguing with their PI and lab mates (and peer reviewers) about the design of studies, which ones will create the story, what can be included in a manuscript, etc. You have sweated bullets over this! And all the reader wants is to see the Figure? AAAAAaaaaaauuuggggghh!

Yeah, you need to get over that.

No Responses Yet to “I really don't give a flying fig about your interpretation of your data”

  1. scicurious Says:

    This is rather what I was hoping to hear. I agree that it’s the stories you form and how well you know your field that matter, not how many citations you can reel off. But I think among trainees there’s an expectation that you need to read a certain amount for it to be “enough”. I have been told that I don’t read “enough”, and I wanted to know what “enough” really meant. For all that it shouldn’t matter how many citations you can reel off in the hallway, the culture kind of promotes it.

    And the data so far indicate that most people read some, and that EVERYONE feels they don’t read “enough”. Which makes me wonder if people feel they don’t read “enough” because they don’t understand their field fully, or if trainees are engaging in chest beating contests.

    Like

  2. gerty-z Says:

    for sure, dude.

    Like

  3. Dr Becca Says:

    Well said, DM.

    Like

  4. Namnezia Says:

    @Sci I think that you are almost better off skimming a lot of papers than “deep reading” a handful. It is more important to have a good lay of the land and get a sense of where the field is as a whole. This is hard to do from the perspective of a few papers. After a while you think mostly of labs or groups of labs and their party line and random experiments (across several papers) that support their party line: “Oh those are the guys that claim X, and X is supported by Z and Y.” So in a sense, its good to *gasp* read review papers carefully to get this perspective, and understand the key experiments in the primary papers. If you want to look up a specific method or control experiment you can then go back to those papers over time.

    Like

  5. drugmonkey Says:

    I have been told that I don’t read “enough”,

    As a mentor, I get very frustrated when I’m talking about papers that form the core background for projects that we are currently working on and trainees are staring at me blankly. Like they haven’t actually read them. And then when I am seemingly always the first to bring up some new, recent paper that pertains. So it is possibly the case that some of this is going on if your mentors are making this comment. It is tricky, however, since the whole damn point of having a multi-scientist group working on a project within a lab is that you will each, through natural behavior, run across different papers that you can bring to the attention of the group. You should not be reading only exactly what the PI or senior postdoc or project lead is reading. But it can get frustrating if you are not reading enough of what everyone else is reading.

    So you might do a self-assessment about whether this is what is going on. Are you reading with intent on a given project? or trying so hard to keep up with “read count” of the latest GlamourMag issues and/or the general subfield of biology that you aren’t actually reading as a scientist (i.e., along specific project-focused lines).

    EVERYONE feels they don’t read “enough”.

    Well sure. Even the goals as I have described them don’t allow you to magically keep up with every single possible pertinent Figure out there. Especially when you are heading a lab with multiple projects with their own literatures. That is why you are relying on your lab group to help out with that. As you descend the ranks, you sort of expect the trainees to be more narrowly focused and therefore to be able to keep up with a greater proportion of the literature on a given project.

    I have noted before that the world’s top expert on a given topic should be the graduate student in the months leading up to the dissertation defense. That is because it is this person who has a singular focus on a topic narrowly enough described that s/he can maintain a pretty active understanding of the vast majority of the related literature.

    Like

  6. scicurious Says:

    Thanks, DM. This is very helpful. For me it’s mostly a problem of my memory being crappy. I do read a ton, but retention…that’s harder. 🙂

    Like

  7. Dr. O Says:

    For me it’s mostly a problem of my memory being crappy. I do read a ton, but retention…that’s harder.

    This is another reason why reading one or a few papers in depth is less preferable to skimming many papers, at least in my own experience. I, too, have an absolutely crappy memory. I won’t remember the fine details of ANY paper, even those I read in depth, a year, maybe even a month from now, unless it is RIGHT up my scientific alley – and even then remembering too many details is a long shot.

    Over the course of my postdoc, I’ve found it much more beneficial to skim through many abstracts on a regular basis (usually on my home page in my RSS reader from PubMed), then click through to the ones that I want more information on, then delve in for a more detailed read when I find it completely relevant to what I am doing right now. Finally, I print out, take notes on, and file away papers that I want to remember for my own work, but know I’ll forget about and not be able to find later. When writing a manuscript or grant, I’ll spend a little bit more time digging online and in my files, but usually with a good *feel* for what’s already out there.

    Otherwise, it’s a lot of energy wasted on invested in only a few papers, and a lot of great literature missed.

    Like

  8. Travis Says:

    Is it common practice for people to distribute relevant papers to colleagues/students/mentors who they think would find them useful? Various members of our team send around papers, news reports, blog posts, etc every week to a general list serve. It’s not uncommon for a senior researcher to send around the table of contents from a journal issue and mention which articles are likely to be of interest to specific students. I’ve come across some really useful papers this way – including ones that have led directly to publications.

    It sure makes it a lot easier for people to stay up to date when you have 10-15 people sending each other relevant papers, having mini-discussions via email, etc, and I think it helps limit the number of blank stares that we students give when the senior researchers are discussing a topic. Even if you don’t have time to meditate on the finer aspects of the papers being sent around, you at least know that they exist and that at least one colleague thinks it is important.

    Like

  9. FSGrad Says:

    @Travis,

    My lab group has made a list of relevant journals to our field and assigned each to a lab member. We’re each responsible for keeping up with that journal and bringing papers of interest to the attention of others. On top of that I do a fair bit of emailing back and forth with friends when I find papers that are probably of interest to them. Both of these exercises have been really useful in helping me keep up with the field. A listserv sounds like it could be very useful as well.

    Like

  10. leigh Says:

    fuck yes. the day it hit me that all i wanted to see were the data and i would gather myowndamnconclusion? that day i gained so much freedom.

    Like


  11. Dude, what the fucken fucke is a “flying fig”?

    Like

  12. Isis the Scientist Says:

    I suspect that people who have to brag about how many papers they are “reading” aren’t actually doing much “writing.”

    Like

  13. brian Says:

    Drugmonkey,
    As I responded to you in Sci’s post, you obviously have misunderstood what I was saying in my original comment. I think the idea of a read list is ridiculous. My point was in agreement with Sci in saying that people try to hard to artificially raise a number on a meaningless variable. But you appear to have taken my comment completely out of context, which I guess means I could have been clearer, or you didn’t read it carefully enough. Not really sure which happened but to clarify, I was saying that to truly read a paper, which was what was being discussed on Sci’s blog, you need to actually understand what the paper says and figures are never going to be enough to get that. I’ve had way too many conversations with people who’ve only looked at the figures that thought a paper was interesting and great, when in actuality the methods were horrendous and invalidate any conclusions that can be drawn from the figures. So I still believe that to say I’ve read paper X, one needs to be able to have an intelligent conversation about the major facets of that paper, and I don’t believe a look at a set of figures will allow you to do that. I actually spoke with a number of post-docs and a few faculty in my department about this yesterday, and while the opinions varied they all agreed that to actually be able to discuss the paper intelligently the figures aren’t going to cut it. I hope this helps to clarify my position.

    Like

  14. drugmonkey Says:

    Oh, I think I got your position about right, brian.

    You don’t seem to grasp, or are disagreeing with, my point that your “truly read” business is just one aspect of the way scientists cover the literature. So trying to limit a definition of “reading” papers in service of quantifying your monthly labors is nonsense.

    “intelligent conversation”? Dude this isn’t the snifter-society of dick jousting, I’m trying to get some science done over here.

    Like

  15. daedalus2u Says:

    This reminds me of three quotes.

    The first two are from Charles de Gaulle:

    When asked how many nuclear weapons France needed, his answer was “enough”. That is how many papers you need to read, “enough”.

    The second one I am paraphrasing: “[Trash bins] are full of indispensable [papers].”

    The third I don’t remember who said it, the question was “when do you know something”, and the answer was “when you can use it”.

    I agree with DM, that the heart and soul of a scientific paper is the data. I too, often don’t bother with the author’s conclusions, or even their hypothesis, just their data. Usually the data is good, even if the hypothesis and analysis is crap. What I really like is going to the literature looking for data on a specific hypothesis I have, finding a paper where the authors had a similar idea, took some data and their data confirms my hypothesis and not theirs. This happens a lot, and tells me that their data is likely pretty good, and that the authors are good and reliable scientists because they published what the data said, and not what they thought it was going to say.

    Since I don’t have a good memory, I often just collect papers, briefly look at the abstract, or even just the title and put them in my files for referring to later, for when I need data which the paper has.

    I agree that it is much better to skim a lot of papers than read a few in depth. You need to acquire your own “sense” of the field, and that only comes from an appreciation of lots of “data” in the field. It is very rare for there to be only singular sources of data about a particular physiological process. When there is only a single source, that result is more open to question than when there are multiple and redundant sources from different researchers looking for different things and finding data that fits perfectly into my conceptualization.

    That is why I am not very concerned with remembering the details of any particular paper. If a paper is valuable enough for me to glance at it, it is valuable enough to have a copy of on my hard drive so I can refer to it later.

    To address Brian, if what you want to do is argue about the minutia details of a hypothesis and conclusions that a specific group of researchers had about their specific data, then, yes, you do need to read the minutia details of their papers and remember them. If you simply want to add to and refine your own conceptualization of the general field which tangentially intersects with that paper, then all you need to know is the data and how it fits into your conceptualization. If it does fit, you don’t need to do much more. If it doesn’t fit, then you need to know and understand why. Is it bad data? Or a bad conceptualization on your part. If it is the latter, then you need to change your conceptualization.

    In order to discuss a paper, I need to have the paper in front of me. I can’t do it from memory.

    Like

  16. DJMH Says:

    I read a lot of abstracts but very few papers…..usually once I’ve seen the abstract, it’s either “Oh yeah, that’s the poster I saw at SfN” or “This paper will bore the pants off of me.”

    But I go to journal clubs regularly to keep me nimble.

    Like

  17. Grumble Says:

    CPP,

    A flying fig refers to the fruit of certain species of ficus, when said fruit is being transported through the air.

    I’m sure “fucken fucke” has a similarly succinct definition, but at least “flying fig” is spelled correctly.

    Like

  18. drugmonkey Says:

    Another thought for those who think they have weak memories..

    Once upon an eon ago I entered a new-for-me area and felt like I needed to show mastery very quickly. I kept a table of the lit with columns for the author and year and then some critical features of design and findings. I referred to this often, added to it often and even whipped it out for reference in discussions with my lab head once or twice.

    You know what? Now I suggest to my trainees that they do the same. I don’t expect people to have photographic memories, I expect them to be able to marshall the evidence into shape when they sit down to write (or to design a study).

    Science is an *open book* test, yo.

    Like

  19. para_sight Says:

    Brian – I also take issue with the idea that looking at figures doesn’t cut it. Actually, if you’re doing it right, sometimes looking at the figures is ALL YOU NEED. Indeed, I know at least one very respectable scientist who builds great papers by deciding what the key figures are going to be that describe the results, and then building the text around that. For many papers, a single figure encapsulates the entire point of the project.

    Like

  20. Spiny Norman Says:

    What papers have I read most closely? In too many cases, the worst ones. The ones that came out in splashy journals, on something really close to what we work on, and turned out to be wrong. We have to know why they were wrong. Maybe we have a result that seems completely contradictory. Maybe we just have a bad feeling. But for our work to progress we have to understand what derailed our competitor’s (or worse, collaborator’s) project.

    The why is almost invariably buried in the details, maybe in the methods or the supplementary data. Were the backcrosses done? How efficient was that IP? Was the antibody from that two-word company whose name is derived from Spanish actually detecting what the catalog claimed it was detecting? Was that inhibitor actually acting through its putative target? Would the results be different in an animal with more heterozygosity?

    To paraphrase DM, the details don’t matter until they matter.

    Like


  21. Fucke looking at the figures. If you are a real player in your field, all you need to look at is the title and the last author, and you will be reminded of the data that you have been privy to already. If data is coming out in your field that you are surprised to see, you need to spend less time reading papers and more time getting yourself plugged in to the behind-the-scences network.

    Note that his applies to PIs, not trainees. Trainees do need to read papers, and the more the better. But it is much more important for trainees to be reading papers *outside* their direct interest zone than inside it. PIs should be reading essentially no papers at all.

    Like

  22. drugmonkey Says:

    Uh-huh. Keep telling yourself that, sporto…

    Like

  23. Ed Yong Says:

    You said “Bollocks”. I’m so proud of you.

    Like

  24. drugmonkey Says:

    For those of a certain age and musical taste, Ed…..

    Like


  25. […] they think they read enough (so far, most respondents don’t think they do). Which prompted this rather peeved reaction from DrugMonkey, about how the number of papers one reads is meaningless, […]

    Like


  26. Wow, remind me never to read this blog again….

    Anxiety about not keeping up with the literature is an age-old concern, and one that SciCurious has every right to address. Derek de Solla Price notes in “Little Science, Big Science” (have you read this?) that even in the early 18th centrury scientists found it hard to keep up with the pace of knowledge production. Slapping down a trainee for having these legitimate anxieties makes me worry about your qualitifications as a “mentor” (and and should do for anyone considering working with you — that is, it should do if you were brave enough to put your name to your writing so people had a chance to vet working with you.)

    The only thing I agree with from your diatribe is the following: “There is only a scorecard for how great your science chops are.” Please post a link to your papers, so the world can evaluate whether to respect your opinion on how to do science? Put up or shut up, I say….

    Yours,
    CMB

    Like

  27. drugmonkey Says:

    If it makes you feel any better my indignant friend, SciCurious knows perfectly well who I am.

    Like


  28. Wow, remind me never to read this blog again….

    YEAH! ME, TOO!!!!!!!!!!!!

    Like

  29. Kaleberg Says:

    A good example of this is the discovery of the neutron. Curie’s team in Paris did the experiment and released the data, but they didn’t realize that they had discovered a roughly proton mass neutral particle. Chadwick’s team in Cambridge looked at the data and recognized the neutron. Of course, they had to duplicate the experiment in a hurry, hoping no one in Paris re-interpreted their data.

    Sometimes it pays to just look at the data and interpret for oneself. It got Chadwick a Nobel Prize.

    Like


  30. […] blogosphere blowhard Physioprof says that he doesn’t have to read papers because he’s so well-connected that as soon as he sees th….  I agree, reading is for […]

    Like

  31. eli rabett Says:

    As Eli said elsewhere

    What trainees lack as a group is perspective, an understanding of how everything fits together and a sense of proportion. Graduate training is designed to pass lore from advisers to students. You learn much about things that didn’t work and therefore were never published [hey Prof. I have a great idea!…Well actually son, we did that back in 06 and wasted two years on it], whose papers to trust, and which to be suspicious of [Hey Prof. here’s a great new paper!… Son, don’t trust that clown.] In short the kind of local knowledge that allows one to cut through the published literature thicket.

    Like


Leave a comment