The "arsenic affair" began with a NASA press conference on Dec. 2, 2010 announcing that a new species of bacteria had been discovered. The species was named GFAJ-1 (Get Felisa a Job), by the lead author Felisa Wolfe-Simon. GFAJ-1 was grown in a medium that lacked phosphate and contained high concentrations of arsenic. The paper, published that day on the Science website, claimed that arsenic was replacing phosphorus in many of the cell's molecules, including nucleic acids.Like many other scientists, I was very skeptical from day one. The results reported in the press conference just couldn't possibly be true unless everything we knew about chemistry and DNA was very wrong.
How did this paper ever get published in Science? I was suspicious that the normal peer review process had been skipped in order to get a major discovery into press as soon as possible.
Turns out that wasn't true. There were three reviews and they were all glowing. We know this because USA Today has obtained copies of the reviews through the Freedom of Information Act in the USA [Glowing reviews on 'arseniclife' spurred NASA's embrace] [Excerpts for the "asreniclife" investigation file]. Here are excerpts from the three reviews—I've never seen such glowing reviews.
Review 1Looks like we can blame the reviewers, or perhaps the editor for choosing the wrong reviewers.
The manuscript Wolfe-Simon et al. demonstrates for the first time that a microorganism is able to use arsenic in place of phosphorus to sustain growth and life. This was done by using a rather simple initial selection on synthetic growth medium followed by a more in-depth analysis of the isolated organism with regard to the path of arsenic from uptake to incorporation into various cellular fractions using ICP-MS, 73As labeling and X-ray absorption near edge spectroscopy (XANES).
The results are exceptional as they show that arsenic, yet believed to be highly toxic for most organisms, in GFAJ-1, a member of the Halomouadaccae, can substitute for the lack of phosphate, a major building block for various macromolecules present in all cells, namely nucleic acids, lipids and proteins.
The methods applied are straightforward. The most surprising and acknowledgeable aspect of the work is its simple approach.
I have only a few minor points regarding the overall presentation.
Review 2
The manuscript by Wolfe-Simon et al. is well-written, concise, to the point and provides exciting novel results. The authors provide many lines of evidence to prove their point that the isolated novel bacterium (at least to some extent) can replace phosphate by arsenic in its biomolecules. It's a pleasure to get a well-conceived and carried-out study to review.
Review 3
Reviewing this paper was a rare pleasure. It is clearly-written and well-reasoned. The authors choose the right methods, designed the right experiments, obtain solid data supporting the conclusion that GSAJ-1 uses As in place of place of P. They use appropriate caution in interpreting results. I think the paper is just about publishable as is; my comments for revision are below. Great job! I look forward to seeing follow-up work in the future.
[Hat Tip: Michael Eisen (@mbeisen) (#arseniclife)]
Erb, T.J., Kiefer, P., Hattendorf, B., Günther, D., and Vorholt, J.A. (2012) GFAJ-1 is an arsenate-resistant, phosphate-dependent organism. Science, 337: 467-470. [doi: 10.1126/science.1218455 ]
Reaves, M.L., Sinha, S., Rabinowitz, J.D., Kruglyak, L., and Redfield, R.J. (2012) Absence of detectable arsenate in DNA from arsenate-grown GFAJ-1 cells. Science, 337: 470-473. [doi: 10.1126/science.1219861]
Wolfe-Simon, F., Switzer Blum, J., Kulp, T.R., Gordon, G.W., Hoeft, S.E., Pett-Ridge, J., Stolz, J.F., Webb, S.M., Weber, P.K., Davies, P.C.W., Anbar, A.D. and Oremland, R.S. (2011) A bacterium that can grow by using arsenic instead of phosphorus. Science. 332:1163-1166. Published online 2 December 2010; published in Science magazine Jun 3, 2011 [doi: 10.1126/science.1197258]
How does FOIA apply to this? Because the first author is (I assume) a federal government employee? Is all correspondence of government employees considered non-private? I find this a bit disturbing.
ReplyDeleteCareful choice of reviewers is a valuable skill. Science lets you nominate reviewers, if I recall, though the editor can ignore your nominations. So maybe these reviews were all by nominated reviewers, and they picked friends.
My understanding is that all professional communications regarding federally funded research are subject to the FOIA. In addition, I've heard, though don't know for certain, that at institutions where grant overhead is used to fund the institute's email servers etc (which is common at soft money institutions), all email accounts at the institute can be subject to FOIA requests.
DeleteWait: You mean all my emails to colleagues are subject to FOIA, during any period I'm on a grant or use an institutional server? That's insane.
DeleteAgain, I don't know this for certain; I work at a non-profit research institution funded primarily by NIH grants, and heard that this was the case in specific reference to our email accounts, which are undoubtedly government funded since most of our administrative budget is covered by grant overhead.
DeleteThat being said, a quick bit of googling doesn't offer any confirmation of this. In principle, I believe FOIA covers all data resulting from federally funded research, including notebooks etc. Then again, there was that big case in Virginia where global warming deniers could not get access to Michael Mann's emails even though he was working at a public university at the time. So not really sure what if any general rules exist...
As for Iron Lisa - she was a NASA fellow, working at USGS, and she now works at LBNL, so her "official" email for professional communications was and is a .gov address, those are definitely subject to FOIA.
That is, by the way, why they only got FWS reply, and not the reviewer reports directly, or, presumably, their names. FOAI wouldn't apply to Science magazine, so only things sent to/from FWS could be requested.
DeleteErr, further simplifying it for this specific case, I just noticed the review document cover letter has an official Department of the Interior letterhead, so presumably an FOIA request was made in regards to USGS
Delete@Harshman:
DeleteWait: You mean all my emails to colleagues are subject to FOIA, during any period I'm on a grant or use an institutional server? That's insane.
You're not far off. All your emails relating to your work, that is, non-personal, are indeed subject to FOIA and anyone in the USA can ask to see them.
If you write a naughty email to your wife/girlfriend from your work email address, it's technically not covered by FOIA. Technically. But the Catch-22 is, if someone makes a FOIA request, then a gov't apparatchik has to go through ALL your emails, sifting out the naughty ones from the ones relevant to your work.
How do you like them apples?
Reviewer 3: "Great job! I look forward to seeing follow-up work in the future."
ReplyDeleteOh, beware what you wish for...
Let's remember that one of the popularizers of Arsenic Life was science journalist (not a scientist) Elizabeth Pennisi at Science, who also pushed the Encode press release Death of Junk DNA hoax.
ReplyDeleteThere is something seriously wrong with News & Views type writing, and peer review might be in trouble too.
Remember that article by Granville Sewell about how 2LOT makes evolution impossible? That got through peer review.
How can we hold idiot peer reviewers accountable?
ReplyDeleteDoubt that you can. Imagine the trouble that would arise if reviewers were somehow liable for incorrect conclusions. Or the spectacle and bother of having an investigation in every such case to see if the reviewers' original judgements were reasonable at the time. The thing is, this sort of situation is common and not rare in science...the only aspects of the Arsenic Affair that set it apart is that the implications of the work were rather profound and that it was hyped up by Science, NASA, and the popular media.
DeleteI recall my greatest skepticism arose from the use of "metalomics" in some of Felicia's writings. Or perhaps that was my greatest nausea. But I must confess that an author who could use proper grammar and wrote clearly might get past me, if only for standing head and shoulders above the typical manuscript I see these days.
ReplyDeleteSeriously? These three reviews combined are shorter than an average single review for an average paper in JBC. That's a hack job on every level. These reviewers need to be publicly shamed.
ReplyDeleteThey need to be shamed, but those weren't the entire reviews, merely the first paragraph of each. So you can fault them on stupidity, but not on length.
DeleteI'm willing to bet that author suggestion of reviewers is a major driving force in such matters. I guess these suggestions make the job of editor easier in that the editor need not investigate themselves who might best review the work.
ReplyDeleteI once had a paper of some midling impact also receive quite glowing reviews (a rarity in my experience to be sure). Afterwards, a fellow post-doc asked me if I had recommended to the editor that my mother and father be reviewers for that manuscript. Jolly joker.
In the wider sense, the way the paper was torn apart afterwards was also peer review. I don't think peer review is broken, nor am I sure how it could be reformed to make it much better. Most ways to improve it appear to include removing the anonymity of reviewers, and that sounds like a very bad idea. There will always be failures, no way around it.
ReplyDeleteI think the problem is less peer review as such but the sensationalism, short-termism and pressure to publish as quickly as possible.
In the wider sense, the way the paper was torn apart afterwards was also peer review.
ReplyDeleteAbsolutely! Peer review does not end with 2-3 people who say the manuscript meets minimum standards for publication.
In a related theme, in the good old day when we had no fear of being deliberately scooped by competitors, we would go to conferences and discuss our data openly, even in the preliminary stage. Imagine all the drama that would have been avoided if she had presented the data before hand at a conference. Most of the concerns that sprung up after publication would probably have been noticed by people in the crowd, and this whole thing could have been avoided.
We need to become a community of peers again, not a community of competitors.
I would not say that the critical response by colleagues on the web and in the literature "was also peer review". That equates "peer review" with criticism in the literature. We're talking about pre-publication review by reviewers who are anonymous to the audience of the paper, and probably anonymous to the author of the paper.
ReplyDeleteAnonymity of peer review is of course important, particularly when a powerful senior person who is potentially nasty has submitted a paper. Then it gives the reviewer some protection.
There have been some experiments with publishing the names of reviewers -- after the fact. The late Lee Van Valen self-published his own journal, Evolutionary Theory where he would ask you, as a reviewer, to consider authorizing him to publish your name in a list of reviewers at the end of the paper ("Reviewed by ...").
Once he asked me to review a paper by one of his students, one that I found to be not very good. His review request included asking me to authorize publication of my name in that list. I wrote a negative review and said also that he could publish my name in the list, provided he put next to it "(recommended rejection)". He published the paper unchanged, but did not list my name. Apparently his support of openness had its limits.
But I digress ...
I have to add my own tangent here. You indirectly brought up editorial choice in this. I think this is an important reason why post-publication discussion is part of the peer review. This is a tangent, and not related to the situation Larry was referring to.
DeleteMy PI had me help him with a review recently for Cell. I had a number of criticisms about the sloppy statistics, and we suggested they be corrected as a minimum for publication, and the paper not go to cell.
A few months later, we get a rebuttal saying that the rejected our request for clearer statistics, but we also saw that 1 other reviewer was very critical of the paper, and 1 loved it. Since they did not take our advise on revising the manuscript, we again said we did not support publication.
Two months later, we get the "paper will be published" notice, thanking us for our comments. We went to the online submission site and checked. In the final round, the editor included a 4th reviewer. 3 of the 4 reviewers did not suggest publication, and the guy who loved it from the start was the only one who thought it was worthy. 3/4 of us did our job, and the editor over-rode us, and in a major journal.
So this is why I feel post-publication peer-review matters.
That equates "peer review" with criticism in the literature.
ReplyDeleteI deliberately wrote "in the wider sense". My main beef was with the idea that peer review is somehow broken. So what would replace it if it were?
Some colleagues I have spoken to have argued that we should replace the entire system with online-only publication in ArXiv-like databases, with peer review taking place in blog-style comment streams below each article, perhaps even with likes or up- and down-votes. Obviously, I have my reservations, but they are right that there are many ways one can design peer review - in the wider sense of the peers judging the quality of scientific work, regardless of the stage of publication.
(The first stage is of course often before any work has been started at all, when the grant proposal is reviewed.)