Monday, January 10, 2011

How to Do Good Science

Richard Feynman (1918–1988) was a very smart American physicist who's words are often quoted ... for good reason.

Here's one quotation where he describes how good scientists should behave. It's a point I make in my class on scientific controversies and it's worth emphasizing because so many modern scientists ignore it.

Feynman is specifically referrring to "cargo cult science" but his advice applies to a lot of of modern biology as well.
There is one feature I notice that is generally missing in "cargo cult science." It's a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty — a kind of leaning over backwards. For example, if you're doing an experiment, you should report everything that you think might make it invalid — not only what you think is right about it; other causes that could possibly explain your results; and things you thought of that you've eliminated by some other experiment, and how they worked — to make sure the other fellow can tell they have been eliminated.

Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can — if you know anything at all wrong, or possibly wrong — to explain it. If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it. There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.

In summary, the idea is to try to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another.


Richard Feynman, "Cargo Cult Science" in Surely You're Joking, Mr. Feynman!
Think about Feynman's words next time you read a paper on the importance of alternative splicing, the disappearance of junk DNA, or anything about evolutionary psychology.

13 comments :

  1. For example, if you're doing an experiment, you should report everything that you think might make it invalid — not only what you think is right about it; other causes that could possibly explain your results

    Reviewers and editors do a fine job making sure none of that happens. Try the above and you will find that in most cases the manuscript will be unpublishable because it reports negative results and "unproven" speculations.

    ReplyDelete
  2. People in biological sciences who think like Feynman will also find their grant applications go unfunded.

    ReplyDelete
  3. I agree with both comments. There are powerful disincentives to thinking and behaving like a proper scientist. Critical thinking is strongly discouraged in today's science culture.

    What does that mean? It means that it's not just individual scientists who are to blame. The entire system is corrupt.

    This isn't news. Almost all scientists I talk to know that much of the scientific literature is bullsh*t.

    The question is, can we do something about it? It's fun to pick on individual scientists who make ridiculous claims but I'm not sure the blogosphere is having any impact on the behavior of most scientists. And aside from blogs, I don't know of any other way of calling attention to the problem.

    ReplyDelete
  4. Exactly. If a scientists fully considers all the likely flaws in his/her hypothesis, he/she probably wouldn't bother to publish the research in the end.

    We have to realize that much of science is interpretative, rather than completely objective.

    Unfortunately, the peer-reviewed literature on evolutionary theory is replete with speculation of what might have rather than on what might not have been the case.

    It is sad to see scientists resorting to sensationalism and spin as a tabloid journalist or politician would, but then science is now a multi-billion dollar industry......but so too is pornography.

    ReplyDelete
  5. LM: The question is, can we do something about it?

    Yes, in theory this is as simple as reworking the reward structure of biomedical science to reward careful, thoughtful, thorough work vs flash-in-the-pan looks-sexy work, both with high impact publications and grant dollars. Since both the awarding of high-impact publications and grant dollars are largely controlled by the scientific community itself through the peer review process, it requires a rework of the basic mindset of the scientists. Unfortunately, hucksterism sells. Once upon a time there was still room at the table for careful science, but in these tight funding times I don't think that is still true.

    ReplyDelete
  6. Reza:
    "Unfortunately, the peer-reviewed literature on evolutionary theory is replete with speculation of what might have rather than on what might not have been the case. "

    How do you suppose the literature in the ID/creationist camp is these days?

    ReplyDelete
  7. I agree with both comments. There are powerful disincentives to thinking and behaving like a proper scientist.

    ReplyDelete
  8. Darwin wrote along a similar path in his autobiography: "The success of the 'Origin' may, I think, be attributed in large part to my having long before written two condensed sketches, and to my having finally abstracted a much larger manuscript, which was itself an abstract. By this means I was enabled to select the more striking facts and conclusions. I had, also, during many years followed a golden rule, namely, that whenever a published fact, a new observation or thought came across me, which was opposed to my general results, to make a memorandum of it without fail and at once; for I had found by experience that such facts and thoughts were far more apt to escape from the memory than favourable ones. Owing to this habit, very few objections were raised against my views which I had not at least noticed and attempted to answer."

    It's a quote I always keep close at hand.

    ReplyDelete
  9. Reza:
    We have to realize that much of science is interpretative, rather than completely objective.

    We also have to realize that eventually, what begins as interpretive in science is either objectively confirmed by the accumulation of foundational evidence or disestablished by it. These unceasing, sly attempts to equate religion with science, science with religion, need to be put into their proper perspective. In science, truth is determined by the explanation that best accords with observation, withstands the potential falsification of its predictions, and yields results. In religion, “truth” is determined ultimately by medieval armies lining up on floodplains and hurling themselves at one another, and accords with the identity and beliefs of those still in possession of the field at the end of the day (the “evidence”).

    ReplyDelete
  10. Would someone please provide a definition of, "evolutionary psychology"?

    ReplyDelete
  11. @Denny

    The wiki entry is at http://en.wikipedia.org/wiki/Evolutionary_psychology

    ReplyDelete
  12. "In summary, the idea is to try to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another."

    And in who's mind is that information interpreted? What is information, and by extension, what is mind? Can mind even be defined by your definition of information?

    ReplyDelete
  13. anonymous asks,

    And in who's mind is that information interpreted? What is information, and by extension, what is mind? Can mind even be defined by your definition of information?

    Let me guess. You're a philosopher, right? Probably a post-modernist.

    ReplyDelete