When I read a new paper I immediately test it against my model of how things are supposed to work. If the conclusions of the paper don't fit with my views, I tend to be very skeptical of the paper. Of course I realize that my model could be wrong and I'm always on the lookout for new results that challenge the current dogma, but, in most cases, if the paper conflicts with current ideas then it's probably flawed.
This is what people mean when they talk about making sense of biology. The ENCODE papers don't make sense, according to my model of how genomes work so I was immediately skeptical of the reported claims. The arseniclife paper conflicted with my understanding of the structure of DNA and how it evolved so I knew it was wrong even before Rosie Redfield pointed out the flaws in the methodology.
What this means in practice is that it requires a great deal of wisdom and experience in order to really make sense of the scientific literature. If that's true, then undergraduates are not in a good position to benefit from the primary scientific literature without the guidance of an experienced instructor. They need a lot more instruction on basic principles and concepts so they can construct, and have confidence in, a good working model of various processes. This goes against the grain of what you find in the pedagogical literature where there's a great deal of emphasis on getting senior undergraduates to experience the primary scientific literature. I'm very skeptical of this emphasis.
All this is by way of introducing a blog post from last September [Transcription Initiation Sites: Do You Think This Is Reasonable?]. I was commenting on a paper by Venters and Pugh (2014) on the number of potential transcription start sites in the human genome. They estimated that there were about 500,000 transcription start sites and most of them would be functional.
This conflicts with my model of the number of genes in the human genome so I assumed that the paper was flawed in spite of the fact that it was published in Nature. I wondered how many other people were skeptical. Not many, according to the comments.
Now a reader, Rui Lopes, tells me that the paper has been retracted!!!! (See his comment on the original post.)
Matthias Siebert and Johannes Söding of the Ludwig-Maximilians-Universität in Munich (München), Germany published a paper criticizing the Vincent and Pugh results (Siebert and Söding, 2014). Here's the abstract ...
How cells locate the regions to initiate transcription is an open question, because core promoter elements (CPEs) are found in only a small fraction of core promoters1, 2, 3, 4. A recent study5 measured 159,117 DNA binding regions of transcription factor IIB (TFIIB) by ChIP-exo (chromatin immunoprecipitation with lambda exonuclease digestion followed by high-throughput sequencing) in human cells, found four degenerate CPEs—upstream and downstream TFIIB recognition elements (BREu and BREd), TATA and initiator element (INR)—in nearly all of them, and concluded that these regions represent sites of transcription initiation marked by universal CPEs. We show that the claimed universality of CPEs is explained by the low specificities of the patterns used and that the same match frequencies are obtained with two negative controls (randomized sequences and scrambled patterns). Our analyses also cast doubt on the biological significance of most of the 150,753 non-messenger-RNA-associated ChIP-exo peaks, 72% of which lie within repetitive regions. There is a Retraction accompanying this Brief Communication Arising by Venters, B. J. & Pugh, B. F. Nature 511, http://dx.doi.org/10.1038/nature13588 (2014).The retraction by Venters and Pugh (2014) says,
We reported the presence of degenerate versions of four well known core promoter elements (BREu, TATA, BREd and INR) at most measured TFIIB binding locations found across the human genome. However, it was brought to our attention by Matthias Siebert and Johannes Söding in the accompanying Brief Communication Arising (Nature 511, E11–E12, http://dx.doi.org/10.1038/nature13587; 2014) that the core-promoter-element analyses that led to this conclusion were not correctly designed. Consequently, the individual core promoter elements were not statistically validated, and therefore there is no evidence of specificity for most reported core-promoter-element locations. To the best of our knowledge, the raw and processed human TFIIB, TBP and Pol II ChIP-exo data are valid, but subject to standard false discovery considerations. We therefore retract the paper. We sincerely apologize for adverse consequences that may have arisen from the error in our analyses.
Siebert, M. and Söding, J. (2014) Universality of core promoter elements? Nature 511:E11–E12 (24 July 2014) [doi:10.1038/nature13587]
Venters, B.J. and Pugh, B.F. (2013) Genomic organization of human transcription initiation complexes. Nature Published online 18 September 2013 [doi: 10.1038/nature12535] [PubMed] [Nature]
Venters, B.J. and Pugh, B.F. (2014) Retraction: Genomic organization of human transcription initiation complexes. Nature, published online July 23, 2014 [doi: 10.1038/nature12535]
What this means in practice is that it requires a great deal of wisdom and experience in order to really make sense of the scientific literature. If that's true, then undergraduates are not in a good position to benefit from the primary scientific literature without the guidance of an experienced instructor.ReplyDelete
But where are students going to *get* this wisdom and experience other than by reading the literature? I don't think anyone is suggesting that students do this on their own with no guidance; even in grad school there are seminar courses where instructors lead grad students in discussion of papers. The problem is when grad students have only been exposed to lectures and textbooks they tend to be more accepting of what they read and don't have the proper critical attitude. Reading papers, especially flawed ones, is a very valuable exercise for students.
Given the relatively small size of these elements, and the flexibility in their spacing, wouldn't you expect by chance a fair number of them in the junk DNA that are otherwise unassocaited with genes or functional RNAs? Granted 500K would be an insane amount to be formed by chance, but there must be some.ReplyDelete
Reading papers, especially flawed ones, is a very valuable exercise for students.ReplyDelete
Not completely sure, but I'm wondering if I recall correctly that Dr. Moran asks his students to find flaws in papers.
This goes against the grain of what you find in the pedagogical literature where there's a great deal of emphasis on getting senior undergraduates to experience the primary scientific literature. I'm very skeptical of this emphasis.ReplyDelete
I still think it is imperative that students begin accessing primary literature by at least year 3. But not because it teaches them critical analytical skills (necessarily). The benefit is that, at least for those who go on to have a career in science, they become increasingly familiar with how scientific results are communicated (e.g. the structure of papers and also the language used in science, which is different than the sort of language they encounter in the popular literature and even textbooks).
An all too common exercise by about year 3 of an undergraduate degree is to assign a paper to a student and ask them to critique it. The student will be quite unable to do this as they lack the deep knowledge required of the topic, and more importantly they will likely have exactly zero experience with most of the techniques used, and therefore will not have the capacity to understand their limitations, why one technique or strategy may be better than another in a specific case, and so on.
So, when assigning primary lit to undergrad students, the goal for the student should be to effectively comprehend the content. These basic comprehension skills improve with continued and increased exposure to the primary literature. Of course it doesnt hurt to remind students not to believe everything they read, but I am doubtful that most undergrads will have much capacity to effectively differentiate between valid and invalid conclusions.
The increased ability to detect flaws generally will have to wait until students begin to gain their own experience designing and conducting experiments and using an ever increasing number of materials and methods under the guidance of much more experienced mentor, usually as graduate students. Through exposure to the primary literature beginning in their undergrad degree, they will at least be able to effectively comprehend a paper's content - a prerequisite to critical analysis.
I will generally agree with Jonathan Badger's sentiment about the importance of getting students started. The problem is, much of what is published is poorly written crap. (get off my lawn).ReplyDelete
Directing students to an introductory set of well written, landmark papers is the proper start. Then you direct them to something rotten and hopefully teach them how to shred it like a pack of wild jackals. Hopefully that's followed by some that are a mixed bag to teach selective criticism. It's a training wheels approach with two main objectives.
1 It is possible, allowed, OK and even good to write well and clearly.
2 Don't believe everything that gets published.