More Recent Comments

Wednesday, February 15, 2023

Wikipedia vs experts and a proposal for "arbitrators"

Wikipedia is a not-for-profit crowdsourced encyclopedia that's open to anybody who wants to contribute. This is both a strength and a weakness but the weaknesses are becoming important in an age of fake news and misinformation. The rules of Wikipedia mean that amateurs can insert any information into science articles as long as it is backed by a reliable source. But "reliable sources" include the popular press and books that may or may not report the scientific consensus accurately. When knowledgeable experts try to correct information, or put it into the proper context, they are often opposed by Wikipedia administrators who have a built-in bias against experts—a bias that's not entirely unjustified but much abused. Consequently, scientists often get frustrated trying to deal with the rules and traditions of Wikipedians because these rules are very different than the standards in the scientific community.

Here's an interesting article by Piotr Konieczny on From Adversaries to Allies? The Uneasy Relationship between Experts and the Wikipedia Community.

Highlights

  • Collaborating with Wikipedia is increasingly common in academia, though barriers remain.
  • Wikipedia’s anti-elitist culture and academia’s anti-amateur culture are still at odds.
  • Many academics remain unaware of core Wikipedia characteristics (its nonprofit status; the growing consensus around the quality of its content).
  • Wikipedia challenges the primacy of scholarly expertise and blurs the line between experts and amateurs. The result is a clash between younger, more accepting academics and older, more suspicious colleagues.
  • A lack of knowledge and incentives, and a fear of ostracism prevent many academics from publicly collaborating with Wikipedia.

The article does a pretty good job of explaining the conflict but, in my opinion, it tilts too far in favor of defending the behavior of Wikipedia administrators who are quick to ban scientists from Wikipedia for not conforming to their, rather arbitrary, rules and culture. It repeats the standard advice to academics that they should learn how to appease Wikipedia amateurs in order to make a contribution.

In summary, the main reason for the difficulties that experts face when it comes to contributing to Wikipedia is that most of them are not experts at using Wikipedia. They assume its process is similar to other processes they are familiar with, such as in academia. This leads to misunderstandings, some of which can have major consequences.

Let's be clear about one thing; the processes that most academics are familiar with include evidence-based reasoning and rational discussion. That's not what we see on Wikipedia so maybe it's the process on Wikipedia that needs to change? Just saying.

I think he's also on the wrong track when he claims that the latest generation of academics are going to be more accepting of Wikipedia bullies and more willing to put up with the propagation of misinformation on Wikipedia for the sake of populism.

The generation of students who grew up with Wikipedia is now entering the ranks of faculty. And a study of digital literacy among librarians found a negative correlation between a librarian’s age and their views on Wikipedia. Likewise, more and more academics are finding it “useful and desirable to publish research results, or even intermediate data, in open repositories,” an attitude that correlates positively with supportive views of Wikipedia. And the endorsements of Wikipedia in peer-reviewed papers by academics and professionals, who readily and even proudly admit to contributing to Wikipedia, are flowing in. In addition to this, we see growing support for Wikipedia at the institutional level. This is exemplified by endorsements from professional associations, some more than a decade old: the Society for Neuroscience (2009), the American Psychological Society (2010), and the American Sociological Association (2011) all support expert contributions to Wikipedia. Overall, it seems safe to conclude that we are witnessing a great thaw in the relationship between Wikipedia and academia. There seems to be a slow but steady growth of a mutual bond that will hopefully grow stronger in the years to come.

In my opinion, it's Wikipedia that needs to be fixed and not academia. A key part of critical thinking is figuring out which sources are reliable and which ones aren't and, like it or not, that requires knowledgeable experts. What Wikipedia really needs is intelligent administrators who have a demonstrated ability to think critically and be skeptical. They don't have to be knowledgeable about everything but they do have to be capable of judging who is right when there's a debate between amateurs and experts over an edit. If they are going to ban someone from Wikipedia, or arbitrate the views of an amateur over some breach of ethics, then they should base their decision on the merits of the discussion and not on arbitrary rules.

Right now there are 910 administrators who have been elected by Wikipedia editors. The easiest way to become an administrator is to be an active editor for several years and to create new articles on some highly specialized topic such as an obscure author or town or some minor historical event. This pretty much rules out most people with a scientific background since most science topics have been covered already and, furthermore, you will have very little opportunity to demonstrate your writing and editing skills by extensively modifying an existing article.

Most administrators are anonymous and their home pages contain no information about their experience outside of Wikipedia.

I think the mechanism for appointing Wikipedia administrators has to be changed and I think their terms should be restricted to five years with no possibility of re-appointment. I think there should be an intermediate level of administration called "arbitration" and arbitrators should be given power over a subset of articles on Wikipedia. For example, The WikiProject on Evolutionary biology has participants who are concerned with articles on evolution, population genetics, and molecular evolution and the WikiProject on Molecular Biology deals with molecular biology and biochemistry. (There are currently no participants in this WikiProject.) I think you should have to join a WikiProject if you want to edit any articles that are claimed by that group but anyone can join just by registering. When the group reaches a minimum number of participants, and the articles are being frequently edited, the group should be given permission to elect an arbitrator who has the powers of an administrator within that group.

Arbitrators can ban participants from a group. They can lead the discussion on the designation, or removal, of Featured articles. They should moderate the discussion on creation and removal of all articles in the WikiProject.1 They should have a limited term but be eligible for re-election.

The only people who are eligible to become full administrators within Wikipedia, are those who have demonstrated the ability to serve as good and fair arbitrators over a number of years and are recommended by the participants in their WikiProject. Only current administrators can vote on the election of new administrators and they should be careful to keep a balance between the four main groups of WikiProjects: Art and culture, Geographical, History and society, and Science, technology, and engineering.


1. One of the major problems in my areas of interest is the proliferation of articles on topics that are already covered elsewhere. The information in those articles often conflicts. For example, in the article on Evolution we learn that the neutral theory has been "largely abandoned" but that "fact" isn't mentioned in the article on Neutral theory of evolution and in the article on Neutral mutation we learn that the predictions of neutral theory have been confirmed.

23 comments :

Mikkel Rumraket Rasmussen said...

One of the most annoying and illogical problems with writing wiki articles on scientific subjects is that you're not allowed to make an inference in a wikipedia article.

If reference 1 says X, and article 2 says Y, you're not allowed to make an inference from X+Y to Z. No, you need to find another article 3 that says Z for you. Even if it contains the exact inference you want to make, you are not allowed to make it in a wiki particle. It must be located in an article you need to cite.

That means wikipedia articles will always just amount to assertions with references.

That's utterly ridiculous.

Larry Moran said...

The Wikipedia police are fond of telling us that you aren't allowed to make inferences or reach conclusions that aren't specifically cited but science articles are littered with such inferences and conclusions.

For example, most Wikipedia editors are firmly convinced that all the old scientists believed that all noncoding DNA was junk based on no evidence whatsoever.

However, when you and I pointed out that no reasonable scientist ever thought that ribosomal RNA genes and regulatory sequences were junk we were told that we needed a specific reference from the 1970s that stated that particular fact.

This is what led to getting me banned from Wikipedia.

Non-coding DNA Talk

Robert Byers said...

There is a greater equation here that affects all. Creationism bumps into it. Indeed who decides who is right, who is a expert, and who may make inferences and all the rest. Wiki, which I think is great, is just revealing a historic problem in science teaching. Its not based on rock hard evidence but on collective opinions of what is accurate. It is getting a vote from selected experts. Like in evolution , geology, etc etc they don't prove thier stuff but consent is made they are right including sincere belief its based on proof. its not. Many ideas in science had thier way before proved whether right or wrong. Wiki is putting a light on human incompetence in investigation and on whom is the boss of truth. The 'science " police bug creationists as much as the wiki police bug everybody.

gert korthof said...

Larry, does wikipedia allow quoting from a (popular-)science book, for example: 'What's in Your Genome'? If so, you can insert the claim that 90% of your genome is junk and referring to that book? Or, are contributors only allowed to quote publications in peer-reviewed scientific journals?

Paul Hutch said...

gert, Yes you can quote a popular science book for a Wikipedia article.

However a person is not allowed to quote their own work in articles. So he'd need to find another respected scientist's book that makes the same claim, or any other person could quote Larry's book.

I can't remember the precise justification for the rule but, when I first encountered it nearly 20 years ago I researched it and ended up understanding why the rule existed.

Piotr Konieczny said...

Piotr here. Few comments - note that I agree with many things you say, and I am not commenting on those.
* it's really hard to get "banned" from Wikipedia. I've seen some academics who gave up after they got annoyed some "random person" reverted their edits, but I can't recall seeing an academic banned. To get banned, one needs to seriously violate the project's rules, by being very combatative/aggressive or engaging in outright harassment.
* "the processes that most academics are familiar with include evidence-based reasoning and rational discussion. That's not what we see on Wikipedia". In my experience this process is common on talk / discusion pages on Wikipedia. In general, discussions there are much higher level than on average Internet forum, and people who behave in a childish/trollish way are ignored or, well, banned.
* "propagation of misinformation on Wikipedia". Recent studies suggest that Wikipedia is pretty good at dealing with misinformation; for example WHO commended Wikipedia on dealing with Covid/vaccines fake news recently.
* "If they are going to ban someone from Wikipedia, or arbitrate the views of an amateur over some breach of ethics, then they should base their decision on the merits of the discussion and not on arbitrary rules." The way the system works is that administrators don't intervene in discussions. Discussions are resolved, indeed, according to the rule of Wikipedia, on the merits on arguments. Administrators only intervene when such discussions degenerate into fights or such.
* "This pretty much rules out most people with a scientific background". Not really. First, you can just edit about your hobby, like fishing or science fiction. Second, many folks become admins without touching content, they just help with maitenance stuff (deletion, vandal fighting, coding, etc.).
* What you describe as "arbitrators" kind of exist (side note: Wikipedia has people called arbitrators, who serve as "judges" and are seen as semi-government of Wikipedia). Anyway, what you describe is known as WikiProject Coordinatos. Although the matrix of roles on Wikipedia is more multi-demensional, for example, Featured Articles are handled by, hmmm, Featured Article Coordinator - who isn't very powerful; they just take care of maintance, kind of like editor in the journal. To get Featured level an article has to pass a Wikipedia peer review process, and then if it passes the FA Coordinator will just do the "bookkeeping" stuff.

Anyway, you are welcome to propose any refors to the Wikipedia system on Wikipedia (I can direct you to the right forum) :)

Larry Moran said...

@Piotr Konieczny

Thank-you for commenting.

I don't agree with some of your comments. For example, some of us have had productive discussions on the Talk pages before making edits but when administrators with no knowledge of the subject get involved the discussions deteriorate rapidly.

That's why the standard advice to academics is to drop the subject entirely as soon as an outside administrator starts lecturing you about Wikipedia culture and rules. That's the signal that rational discussion is over.

In my field, Wikipedia is horrible at dealing with misinformation as soon as an administrator becomes involved. As long as the misinformation has a "reliable" source they will prevent you from deleting it or contradicting it.

You said, "The way the system works is that administrators don't intervene in discussions. Discussions are resolved, indeed, according to the rule of Wikipedia, on the merits on arguments. Administrators only intervene when such discussions degenerate into fights or such.

That's simply incorrect in my experience. I have lots of examples of administrators intervening to revert edits with no explanation.

I agree that WikiProject Coordinators are similar to the arbitrators that I propose. I just want them to be given more power to ban people from editing certain articles and, more importantly, to serve as a stepping stone to becoming an administrator. Administrators need to be able to demonstrate that they can handle disputes and arbitrate conflicts.

Right now the easiest way to become an administrator is to write some articles about your favorite hobby. The really tough part about editing on Wikipedia is handling some very popular articles such as the ones on Evolution or human genome. Those are the articles that scientists are interested in.

T. Finch PhD said...

You complain that people on Wikipedia don't value experts enough, and yet you're "disagreeing" with Piotr, who is an established expert on Wikipedia, about how Wikipedia works. It's pretty funny, really. You want people to value your expertise, but you reject theirs.

I'm something of an expert on Wikipedia, too -- I have nearly 100K edits and have been an administrator for over ten years -- and I can tell you: when it comes to Wikipedia, you really don't know what you're talking about. I'm sure you're highly knowledgeable in your field of expertise, but Wikipedia is not your field of expertise.

T. Finch PhD said...

@Paul Hutch: it's not that you cannot cite your own work. It's that if you cite yourself rather than others when the work of others is just as valid to support the assertion you're making, people might suspect you're doing it for self-serving reasons. So when you can find someone else to cite to, you're better off doing that as often as you can and only citing to your own work when actually necessary.

T. Finch PhD said...

@Mikkel Rumraket Rasmussen, this is a common frustration among academics, who are trained and incentivized to use primary sources to do original research and draw new inferences from that research.

Wikipedia is the exact opposite. Wikipedia doesn't want editors -- who may be complete laymen, no matter what they claim about themselves on their user page -- drawing new conclusions and making new inferences. Instead Wikipedia wants editors to report what inferences actual experts have made. Really not illogical at all.

T. Finch PhD said...

@Mikkel Rumraket Rasmussen you complain that "That means wikipedia articles will always just amount to assertions with references."

That's the entire point of Wikipedia: it's a collection of reliable sources discussing a particular subject, along with prose -- the article itself -- which summarizes what those sources say.

Ideally the sources are curated and the prose competently written. But the point of a Wikipedia article is to summarize what is currently known about a subject, with that summary supported by a list of sources. That's it. That's literally the entire mission.

Larry Moran said...

@T. Finch

I think you need to read carefully what critics are saying. Most of us have a pretty good understanding of the Wikipedia culture. We are arguing that the culture needs to change.

It's very common for Wikipedians to lecture us on not understanding Wikipedia and how it works when that's not the issue at all. In fact, that very response is part of the problem.

If you've been an administrator for ten years then you surely know about the conflict between Wikipedia as an encyclopedia - where facts and accuracy should be the most important features - and Wikipedia as a crowd-sourced social network where policing behavior is the highest priority.

Do you even care about facts and accuracy? Do you have any experience editing popular and controversial science articles like "Evolution" or "Non-coding DNA"? (I can't find "T. Finch PhD on Wikipedia but I only did a quick search.)

T. Finch PhD said...

Yes, I care very much about facts and accuracy, but I also understand how Wikipedia tries to ensure articles are factual and accurate. It's not by saying, "Oh, here's Larry Moran, he says he's a scientist and we can trust him. Okay! Whatever he says goes!" It's by requiring all editors to support all assertions with citations to reliable sources.

I don't actually think you have a good understanding of Wikipedia culture. You are making basic misstatements about it. For instance you dispute Piotr's statement that "The way the system works is that administrators don't intervene in discussions. Discussions are resolved, indeed, according to the rule of Wikipedia, on the merits on arguments. Administrators only intervene when such discussions degenerate into fights or such." Which is absolutely correct. You say " I have lots of examples of administrators intervening to revert edits with no explanation.". Provide those examples. Administrators may, in their capacity as editors, revert edits. If they're not good editors, they might do it with no explanation -- that's frowned upon as not-best-practices, but some editors don't follow best practices. But the only time an admin should be reverting an edit IN THEIR CAPACITY AS AN ADMIN is when they are reverting to whatever their best guess is for a stable version after an edit war. That is, no admin should be reverting an edit BECAUSE THEY DISAGREE WITH THE CONTENT OF THE EDIT. And if you think it's happening regularly, you do not understand Wikipedia administrator conduct. An administrator who, in their admin capacity -- that is, through protection, for instance -- reverts an edit BECAUSE THEY DISAGREE WITH THE CONTENT should be brought to a review board. Any admin who did it regularly would be stripped of their tools. I literally have never seen an admin use their tools in a content dispute more than once, and it's usually in the first months of their adminship when they are inexperienced and the situation is borderline.

T. Finch PhD said...

"If you've been an administrator for ten years then you surely know about the conflict between Wikipedia as an encyclopedia - where facts and accuracy should be the most important features - and Wikipedia as a crowd-sourced social network where policing behavior is the highest priority." Behavior is important as a means to an end -- collaborative work requires civility, for instance. Edit warring wastes time and energy of volunteers. Personal attacks distract from the project. So, yes, we have certain required behavior standards for editors, and if you can't comply, you'll end up with your editing privileges revoked until you convince someone you're going to comply in future.

Facts and accuracy are absolutely important, and the way Wikipedia ensures accuracy is to require sourcing for all assertions rather than simply taking the word of someone who comes in and says, "I'm an expert". Anyone can say that. Anyone can say their name is T. Finch PhD, a Wikipedia administrator of ten years who has 100K edits. Why would you take my word for it? For the same reason, Wikipedia doesn't simply assume whatever Larry Moran asserts is true should be believed without proof. It shouldn't take another scientist to come in and say, "Yes, he's right." You should be able to prove it to any reasonably intelligent, well-educated editor.

Larry Moran said...

@T. Finch

You really like to quibble don't you?

Okay, I admit that I could have phrased my comment more accurately by saying, "I have lots of examples of Wikipedia administrators, who have the power to block and ban me, jumping into content disputes and reverting edits while acting as simple editors."

I will continue to advise people that whenever a known administrator reverts one of your edits and starts disputing the content of your proposed edits, then you should back off and drop the subject. Wikipedia administrators are a special class of Wikipedians with their own agenda and biases and they are quick to anger and generally don't tolerate anyone who disagrees with them.

You also said, "Wikipedia doesn't simply assume whatever Larry Moran asserts is true should be believed without proof." Really? Do you honestly believe that any of us are making that kind of argument? Pay attention if you want to be taken seriously.

You also said, "You should be able to prove it to any reasonably intelligent, well-educated editor." I agree. The problem is that those kind of editors are not common and some of them go on to become administrators.

T. Finch PhD said...

You said 'I could have phrased my comment more accurately by saying, "I have lots of examples of Wikipedia administrators, who have the power to block and ban me, jumping into content disputes and reverting edits while acting as simple editors." ' But that's exactly my point: your beef isn't with admins. Administrators have no power in any content dispute. Your issue is your ability to convince other editors. Which may be very frustrating, of course. But it has nothing to do with admins.

That's not quibbling, that's just an attempt to clear up what has appeared to be a misconception about what admins can and can't do. If you have a content dispute with an admin, they cannot block you at that dispute. Literally the only thing they can do is warn you or complain, just like any other editor can, about your behavior. If you (or anyone, including admins) go to the administrators' noticeboard with a content dispute, they'll tell you to go away because admins don't deal with content disputes. Literally they only deal with behavior at ANI.

My experience is that most people who edit regularly are pretty intelligent -- they're typically curious people who enjoy research and writing -- and I've encountered very few admins who weren't. And mostly they aren't quick to anger, although I'm sure many editors find it annoying when someone is determined to think the worst of all Wikipedia editors, especially when they're fairly inexperienced themselves. It's kind of like the freshman in the front row of the 101 class explaining why you're wrong about some basic facet of the subject you've spent decades working on.

Mikkel Rumraket Rasmussen said...

T. Finch says "Instead Wikipedia wants editors to report what inferences actual experts have made. Really not illogical at all."

Yeah the problem is Wikipedia has no way to determine who's actual experts. All it knows is there's some sort of doi-listed article referenced, with some sort of author, who states something in text form. And there might be issues with that that have not been explicitly stated in the peer-reviewed literature. The data to show this issue might exist in the peer-revied literature, but the data is in a paper where the authors do not explicitly point this out(the paper might have another focus and thus do not contain every imaginable entailment stated in obvious literal form).
So you're now in a situation where some assertion is made somewhere, based on nothing at all, but it's in a doi-listed article. Hence by Wiki rules it basically "expert testimony"(I can find doi-listed crap written by creationists, that doesn't explicitly argue creationism, but nevertheless implies it, and someone could reference that on wikipedia and I would be powerless to correct it because the specific crap stated in that paper has not been explicitly rebutted elsewhere in the literature), and people with too much sense of authority who just likes to push rules are not even willing to listen to explanations why the Wiki article should be corrected because it's pushing misinformation, because well look it's got a doi-listed reference that contains the same assertion as the Wiki article.

Please do not pretend this isn't a problem.

T. Finch PhD said...

MRR, I don't think anyone is pretending there isn't junk science out there or that there aren't editors who try to insert it into articles. But there is an entire WikiProject devoted to what constitutes reliable sources for medicine-related articles. There's a WikiProject Scepticism, a fringe theories noticeboard, a Wikiproject Pseudoscience...the list goes on and on. There are multiple places to go for help when someone pushes incorrect crap.

Larry Moran said...

@Mikkel

It's a shame that administrators like T. Finch don't understand the problem. He thinks it's about "junk science" and that it's easy to recognize it and deal with it.

He doesn't understand that that there real disputes among scientists and that some of these are overblown in the popular press. This means that there's often a scientific discussion about which version of a dispute needs to be covered in a Wikipedia article. He doesn't understand that there are many issues in science articles that need the attention of people who are very knowledgeable about the subject and not just Wikipedians who can scour the literature for confirmation bias.

We can't fix Wikipedia unless a large group of scientifically literate administrators recognize the problem in identifying accurate reliable sources instead of just citing rules of behavior that were made up 20 years ago.

Huinca said...

What an interesting discussion. Funny how Larry purports to be highly skeptical, except when it comes to his own, deeply entrenched beliefs! Then it's Taliban level of dogmatism, and anything goes, even anecdotal evidence ("I have lots of examples..." yeah, right).

Wikipedia is a formidable source of general knowledge for a wide audience, and certainly generally more reliable than expert literature like Nature, which has an immensely biased editorial policy with highly politically charged articles ("The memory of water" by Jean-Jacques Benveniste was work by an 'expert' as is the farcical 'Proximal origin of Covid-19'). Not to mention the $11,000 publication fee that will keep the proletarian researchers (like yours faithfully) at a safe distance.

My point is that statistically speaking (millions of articles, not the three or four that Larry got into fights over), lay people *tend* to get it right as frequently as experts. Experts usually get it wrong too, and sometimes spectacularly so. Do we need examples for that?

So, if anything, 'experts' need to get their own house in order first, how reliable is the current peer review system to claim authoritatively that a 'peer-reviewed article' is more reliable than a random Wikipedia article? We all know the answer to that.

Anyway thanks for the space.

Joe Felsenstein said...

Huinca said: lay people *tend* to get it right as frequently as experts. Experts usually get it wrong too,
Wow, that's a pretty sweeping statement. If taken seriously, it would lead Huinca to advocate the replacement of all experts in, well, anything.

Larry Moran said...

@Joe

Huinca is reflecting the common view of Wikipedians. This is why scientists usually have such a hard time on Wikipedia. Their opinions on scientific issues are actively distrusted on the grounds that scientists are always biased and eager to promote their own original research.

Huica probably believes that all the textbooks in physics, astronomy, geology, chemistry, and biology could be written by lay people. Watch out, Joe, your book is about to get some competition from someone sitting in front of a laptop in their mother's basement!

Joe Felsenstein said...

your book is about to get some competition from someone sitting in front of a laptop in their mother's basement!I'm ahead of them, at least in volume of typing.