Reliability of Wikipedia

The reliability of Wikipedia as an encyclopedia is often assessed in several ways, including statistically, by comparative review, and by analysis of the historical patterns, strengths and weaknesses inherent in the Wikipedia process.

Because Wikipedia is a wiki, and open to collaborative editing by anyone, assessing its reliability requires also examining its ability to detect and rapidly remove false or misleading information.

Several studies suggest that the science entries of Wikipedia are of a similar order of accuracy (and similar rates of both serious and minor errors) to Encyclopædia Britannica, that it provides a good starting point for research, and that articles are in general reasonably sound. However, it does suffer from omissions and inaccuracies and sometimes these can be serious. A separate study suggests that in many cases, vandalism is reverted fairly quickly, but this does not always happen.

Areas of reliability

The reliability of Wikipedia as a whole is a factor of several criteria:

• Accuracy of information provided within articles
• Comprehensiveness, scope and coverage within articles and in the range of articles
• Susceptibility to, and exclusion and removal of, false information (a criterion specific to the Wikipedia process)
• Susceptibility to editorial and systemic bias
• Identification of reputable third-party sources as citations

The first three of these have been the subjects of various studies of the project, while the presence of bias is strongly disputed on both sides, and the prevalence and quality of citations can be tested within Wikipedia.

The Wikipedia editing model

Main page: Wikipedia:Editorial oversight and control

The Wikipedia model allows anyone to edit, and relies on a large number of well-intentioned editors to overcome issues raised by a smaller number of problematic editors. It is inherent in Wikipedia's editing model that while poor information can be added, over time editors reach strong consensus, and quality should improve in a form of group learning, so that substandard edits will very rapidly be removed. This assumption is still being tested, and its limitations and reliability are not yet a settled matter – Wikipedia is a pioneer in communal knowledge building of this kind. This contrasts with many more traditional models of knowledge and publishing, which attempt to limit content creation to a relatively small group of approved editors in order to exercise strong hierarchical control. Theories of critical pedagogy argue that consensus alone maintains the status-quo; new knowledges only emerge from a dialectical exposure of power structures.

Wikipedia's model of knowledge creation is relatively novel, since widespread collaborative projects of its kind were rare until the arrival of the Internet, and are still rare on such a large scale. Over time, Wikipedia has developed many editorial tools that have been found to be useful, based largely upon trial and error.

While Wikipedia has the potential for extremely rapid growth and harnesses an entire community – much in the same way as other communal projects such as Linux – it goes further in trusting the same community to self-regulate and become more proficient at quality control. Wikipedia has harnessed the work of millions of people to produce the world's largest knowledge-based site along with software to support it, resulting in over six million articles written in about six years. For this reason, there has been considerable interest in the project both academically and from diverse fields such as information technology, business, project management, knowledge acquisition, software programming, other collaborative projects and sociology, to explore whether the Wikipedia model can produce good results, what collaboration in this way can reveal about people, and whether the scale of involvement can overcome the obstacles of individual limitations and poor editorship which would otherwise arise.

Another reason for inquiry is the growing and widespread reliance on Wikipedia by both websites and individuals using it as a source of information, and concerns over such a major source being susceptible to rapid change – including the introduction of misinformation at whim. The proponents of such concerns tend to seek reassurance of the quality and reliability of articles, and the degree of usefulness, misinformation or vandalism which should be expected, in order to decide what reliance to place upon them.

Assessment

Accuracy of articles

Reliability of information can be assessed by comparison of Wikipedia articles to their parallel articles in other reputable sources.

A common source of reliability criticisms is the open process involved, which means that any article can be modified for better or worse at any time, and the fact that no privileged versions of articles currently exist in the main encyclopedia. This fluidity has been assessed by specialists both positively and negatively, as has Wikipedia's model that focuses upon rapid correction rather than initial accuracy.

Comparative studies

On October 24, 2005, The Guardian published an article entitled "Can you trust Wikipedia?" where a panel of experts were asked to critically review seven entries related to their fields. One article was deemed to have made "every value judgement... wrong", the others receiving marks from 5 to 8 out of a notional ten. Of the other six articles reviewed and critiqued, the most common criticisms were:

1. Poor prose, or ease-of-reading issues (3 mentions)
2. Omissions or inaccuracies, often small but including key omissions in some articles (3 mentions)
3. Poor balance, with less important areas being given more attention and vice versa (1 mention)

The most common praises were:

1. Factually sound and correct, no glaring inaccuracies (4 mentions)
2. Much useful information, including well selected links, making it possible to "access much information quickly" (3 mentions)

Nature reported in 2005 that science articles in Wikipedia were comparable in accuracy to those in Encyclopædia Britannica. Out of 42 articles, only 4 serious errors were found in Wikipedia, and 3 were found in Encyclopædia Britannica, although more than a hundred lesser errors and omissions were found in each and Wikipedia's articles were often "poorly structured". On March 24, 2006, Britannica provided a rebuttal of this article, labeling it "fatally flawed", to which Nature responded. However, Kister's Best Encyclopedias 2nd edition (1994) compared the accuracy of Britannica to several other encyclopedias, and concludes that although more accurate than many, it is being ranked lower than encyclopedias such as Encyclopedia Americana, World Book Encyclopedia, and Compton's Encyclopedia.

A web-based survey conducted from December 2005 to May 2006 assessed the "accuracy and completeness of Wikipedia articles". Fifty people (a fairly low response rate) accepted an invitation to assess an article. Of the fifty, thirty-eight (76%) agreed or strongly agreed that the Wikipedia article was accurate, and twenty-three (46%) agreed or strongly agreed that it was complete. Eighteen people compared the article they reviewed to the article on the same topic in the Encyclopædia Britannica. Opinions on accuracy were almost equal between the two encyclopedias (6 favoring Britannica, 7 favoring Wikipedia, 5 stating they were equal), and eleven (61%) found Wikipedia somewhat or substantially more complete, compared to seven (39%) for Britannica. The survey did not attempt random selection of the participants, and it is not clear how the participants were invited.

The German computing magazine c't performed a comparison of Brockhaus Multimedial, Microsoft Encarta, and Wikipedia in October 2004: Experts evaluated 66 articles in various fields. In overall score, Wikipedia was rated 3.6 out of 5 points ("B-").

Viewing Wikipedia as fitting the economists' definition of a perfectly competitive marketplace of ideas, George Bragues (University of Guelph-Humber), examined Wikipedia's articles on seven top Western philosophers: Aristotle, Plato, Immanuel Kant, Rene Descartes, Georg W.F. Hegel, Thomas Aquinas, and John Locke. Wikipedia's articles were compared to a consensus list of themes culled from 4 reference works in philosophy. Bragues found that, on average, Wikipedia's articles only covered 52% of consensus themes. No errors were found, though there were significant omissions "[5]

PC Pro magazine (August 2007) asked experts to compare 4 articles (a small sample) in their scientific fields between Wikipedia, Britannica and Encarta. In each case Wikipedia was described as "largely sound", "well handled", "performs well", "good for the bare facts" and "broadly accurate". One article had "a marked deterioration towards the end" whilst another had "clearer and more elegant" writing, a third was assessed as less well written but better detailed than its competitors, and a fourth was "of more benefit to the serious student than its Encarta or Britannica equivalents". No serious errors were noted in Wikipedia articles, whereas serious errors were noted in one Encarta and one Britannica article.

Subjective expert opinion

Librarian views

A 2006 review of Wikipedia by LibraryJournal.com, using a panel of librarians, "the toughest critics of reference materials, whatever their format", asked "long standing reviewers" to evaluate three areas of Wikipedia (popular culture, current affairs, and science), and concluded: "While there are still reasons to proceed with caution when using a resource that takes pride in limited professional management, many encouraging signs suggest that (at least for now) Wikipedia may be granted the librarian’s seal of approval". A reviewer who "decided to explore controversial historical and current events, hoping to find glaring abuses" concluded "I was pleased by Wikipedia’s objective presentation of controversial subjects" but that "as with much information floating around in cyberspace, a healthy degree of skepticism and skill at winnowing fact from opinion are required." Other reviewers noted that there is "much variation" but "good content abounds".

The library at Trent University, Ontario, Canada states of Wikipedia that many articles are "long and comprehensive", but that there is "a lot of room for misinformation and bias [and] a lot of variability in both the quality and depth of articles". It adds that Wikipedia has advantages and limitations, that it has "excellent coverage of technical topics" and articles are "often added quickly and, as a result, coverage of current events is quite good", comparing this to traditional sources which are unable to achieve this task. It concludes that depending upon the need, one should think critically and assess the appropriateness of one's sources, "whether you are looking for fact or opinion, how in-depth you want to be as you explore a topic, the importance of reliability and accuracy, and the importance of timely or recent information", and adds that Wikipedia can be used in any event as a "starting point".

An article for the Canadian Library Association (CLA) discusses the Wikipedia approach, process and outcome in depth, commenting for example that in controversial topics, "what is most remarkable is that the two sides actually engaged each other and negotiated a version of the article that both can more or less live with". The author comments that:

[I]n fact Wikipedia has more institutional structure than at first appears. Some 800 experienced users are designated as administrators, with special powers of binding and loosing: they can protect and unprotect, delete and undelete and revert articles, and block and unblock users. They are expected to use their powers in a neutral way, forming and implementing the consensus of the community. The effect of their intervention shows in the discussion pages of most contentious articles. Wikipedia has survived this long because it is easier to reverse vandalism than it is to commit it..."

Information Today (March 2006) cites librarian Nancy O’Neill (principal librarian for Reference Services at the Santa Monica Public Library System) as saying that "there is a good deal of skepticism about Wikipedia in the library community" but that "she also admits cheerfully that Wikipedia makes a good starting place for a search. You get terminology, names, and a feel for the subject."

PC Pro (August 2007) cites the head of the European and American Collection at the British Library, Stephen Bury, as stating "Wikipedia is potentially a good thing - it provides a speedier response to new events, and to new evidence on old items". The article concludes: "For [Bury], the problem isn't so much the reliability of Wikipedia's content so much as the way in which it's used. "It's already become the first port of call for the researcher", Bury says, before noting that this is "not necessarily problematic except when they go no further". According to Bury, the trick to using Wikipedia is to understand that "just because it's in an encyclopedia (free, web or printed) doesn't mean it's true. Ask for evidence .. and contribute."

In a 2004 interview with The Guardian, self-described information specialist and internet consultant Philip Bradley said that he would not use Wikipedia and was "not aware of a single librarian who would. The main problem is the lack of authority. With printed publications, the publishers have to ensure that their data are reliable, as their livelihood depends on it. But with something like this, all that goes out the window."

Academia

Academic circles have not been exclusively dismissive of Wikipedia as a reference. Wikipedia articles have been referenced in "enhanced perspectives" provided on-line in Science. The first of these perspectives to provide a hyperlink to Wikipedia was "A White Collar Protein Senses Blue Light", and dozens of enhanced perspectives have provided such links since then. However, these links are offered as background sources for the reader, not as sources used by the writer, and the "enhanced perspectives" are not intended to serve as reference material themselves.

An empirical study conducted in 2006 by a Nottingham University Business School lecturer in Information Systems, the subject of a review on the technical website Ars Technica, involving 55 academics asked to review specific Wikipedia articles that either were in their expert field (group 1) or chosen at random (group 2), concluded that "The experts found Wikipedia’s articles to be more credible than the non–experts. This suggests that the accuracy of Wikipedia is high. However, the results should not be seen as support for Wikipedia as a totally reliable resource as, according to the experts, 13 percent of the articles contain mistakes [10% of experts reporting factual errors of unspecified degree, 3% reporting spelling errors]".

The Gould Library at Carleton College in Minnesota has a page describing the use of Wikipedia in academia. It asserts that "Wikipedia is without question a valuable and informative resource", but that "there is an inherent lack of reliability and stability" to its articles, again drawing attention to similar advantages and limitations as other sources. As with other reviews it comments that one should assess one's sources and what is desired from them, and that "Wikipedia may be an appropriate resource for some assignments, but not for others". It cited Jimmy Wales' view that Wikipedia may not be an ideal as a source for all academic uses, and (as with other sources) suggests that at the least, one strength of Wikipedia is that it provides a good starting point for current information on a very wide range of topics.

The academic world's view of Wikipedia has improved during the last few years, as it can be inferred from the increase in the number of citations in international scientific journals. As of July 17, 2007 a search in the ScienceDirect (login required) database (a large online collection of published scientific research produced by Elsevier) for academic and scientific articles citing Wikipedia yields the following result:

Year article published	No. articles citing Wikipedia
before 2003	0
2003	1
2004	9
2005	28
2006	131
2007 (as of July 17)	256

Editors of other encyclopedias

In a 2004 piece called "The Faith-Based Encyclopedia," former Britannica editor Robert McHenry criticized the wiki approach, writing:

[H]owever closely a Wikipedia article may at some point in its life attain to reliability, it is forever open to the uninformed or semiliterate meddler… The user who visits Wikipedia to learn about some subject, to confirm some matter of fact, is rather in the position of a visitor to a public restroom. It may be obviously dirty, so that he knows to exercise great care, or it may seem fairly clean, so that he may be lulled into a false sense of security. What he certainly does not know is who has used the facilities before him." Similarly, Encyclopædia Britannica's executive editor, Ted Pappas, was quoted in The Guardian as saying: "The premise of Wikipedia is that continuous improvement will lead to perfection. That premise is completely unproven."

Other

Information Today (March 2006) comments on Wikipedia and Britannica that it is comparing "Apples and Oranges" and that:

[E]ven the revered Encyclopaedia Britannica is riddled with errors, not to mention the subtle yet pervasive biases of individual subjectivity and corporate correctness... There is no one perfect way. Britannica seems to claim that there is. Wikipedia acknowledges there’s no such thing. Librarians and information professionals have always known this. That’s why we always consult multiple sources and counsel our users to do the same.

BBC technology specialist Bill Thompson wrote that "Most Wikipedia entries are written and submitted in good faith, and we should not let the contentious areas such as politics, religion or biography shape our view of the project as a whole", that it forms a good starting point for serious research but that:

No information source is guaranteed to be accurate, and we should not place complete faith in something which can so easily be undermined through malice or ignorance... That does not devalue the project entirely, it just means that we should be sceptical about Wikipedia entries as a primary source of information... It is the same with search engine results. Just because something comes up in the top 10 on MSN Search or Google does not automatically give it credibility or vouch for its accuracy or importance." He adds the observation that since most popular online sources are inherently unreliable in this way, one byproduct of the information age is a wiser audience who are learning to check information rather than take it on faith due to its source, leading to "a better sense of how to evaluate information sources".

Newspapers: In an edict issued to staff members of the Milwaukee Journal Sentinel sports department on May 31, 2007, assistant sports editor Karl Svatek stated:

A reminder that under no circumstances are we to check facts by referring to Wikipedia. The content on that site simply cannot be trusted.

Removal of false information

Perhaps the most notorious test of false information was the John Seigenthaler Sr. controversy in 2005, when a biography of a famous writer and journalist was found to contain libelous hoax material that had gone undetected for more than four months.

In an informal media test of Wikipedia's ability to detect misinformation, an anonymous blogger tested Wikipedia by inserting subtly erroneous facts into obscure articles, stating that its process "isn't really a fact-checking mechanism so much as a voting mechanism", and that material which did not appear "blatantly false" may be accepted as true. Wikipedians by and large responded with anger at what was considered by many to be an unfair trial which had deliberately focused on obscure, less-reviewed articles; the blogger responded that the test was fair.

Viégas, Wattenberg, and Dave (2004) studied the flow of editing in the Wikipedia model, with emphasis on breaks in flow (from vandalism or substantial rewrites), showing the dynamic flow of material over time. They found that most acts of vandalism during May 2003 were repaired within minutes. However, it is unclear whether or not this finding applies to all forms of vandalism, including so-called 'sneaky' vandalism (which resembles genuine editing and is by nature harder to detect). Lih (2004) compared articles before and after they were mentioned in the press, and found that external referencing of articles incentivized editors to higher quality work. A 2002 study by IBM found that most vandalism on the English Wikipedia was reverted within five minutes, though some persisted for much longer:

We've examined many pages on Wikipedia that treat controversial topics, and have discovered that most have, in fact, been vandalized at some point in their history. But we've also found that vandalism is usually repaired extremely quickly--so quickly that most users will never see its effects."

Misinformation in positive forms may be harder to detect than vandalism, or to distinguish from internet trolling. On March 2, 2007, MSNBC.com reported that Hillary Rodham Clinton had been incorrectly listed for 20 months in her Wikipedia biography as valedictorian of her class of 1969 at Wellesley College. (Hillary Rodham was not the valedictorian, though she did speak at commencement, giving rise to the inaccuracy.) The MSNBC article included a link to the Wikipedia edit, in which user LukeTH added the incorrect information on July 9, 2005. After the MSNBC report, the inaccurate information was removed the same day, with this edit. Between the two edits, the wrong information had stayed in the Clinton article while it was edited more than 4,800 times over the 20 months.

A further informal assessment by the popular IT magazine "PC Pro", for its 2007 article Wikipedia Uncovered tested Wikipedia by a similar device to those described above, by introducing 10 errors that "varied between bleeding obvious and deftly subtle" into articles (the researchers later put right the articles it had edited). Labeling the results "impressive" it noted that all but one was noted and fixed within the hour, and that "the Wikipedians tools and know-how were just too much for our team". A second series of another 10 tests, using "far more subtle errors" and additional techniques to conceal their nature, met similar results: "despite our stealth attempts the vast majority ... were discovered remarkably quickly... the ridiculously minor Jesse James error was corrected within a minute and a very slight change to Queen Ann's entry was put right within two minutes." Two of the latter series were not detected. The article concluded that "Wikipedia corrects the vast majority of errors within minutes, but if they're not spotted within the first day the chances ...dwindle as you're then relying on someone to spot the errors while reading the article rather than reviewing the edits."

Coverage

Wikipedia has been accused of deficiencies in comprehensiveness because of its voluntary nature, and of reflecting the systemic biases of its contributors. Encyclopædia Britannica editor-in-chief Dale Hoiberg has argued that "people write of things they're interested in, and so many subjects don't get covered; and news events get covered in great detail. The entry on Hurricane Frances was five times the length of that on Chinese art, and the entry on Coronation Street was twice as long as the article on Tony Blair." (As of December 2005, this is no longer the case.) Former Nupedia editor-in-chief Larry Sanger stated in 2004, "when it comes to relatively specialized topics (outside of the interests of most of the contributors), the project's credibility is very uneven."

Wikipedia has been praised for making it possible for articles to be updated or created in response to current events. For example, the then-new article on the 2004 Indian Ocean earthquake on its English edition was cited often by the press shortly after the incident. Its editors have also argued that, as a website, Wikipedia is able to include articles on a greater number of subjects than print encyclopedias may. The New York Times, noted that Wikipedia served as an "essential news source" on the Internet in the aftermath of the shootings at Virginia Tech University. Even The Roanoke Times, which is published near Blacksburg, Va., where the university is located, noted that Wikipedia "has emerged as the clearinghouse for detailed information on the event."

Broad assessments

Several commentators have drawn a middle ground, asserting that the project contains much valuable knowledge and has some reliability, even if the degree is not yet assessed with certainty. Many of the librarian and academic reviewers of Wikipedia cited above take such a view.

Others taking this view include Danah Boyd, who in 2005 discussed Wikipedia as an academic source, concluding that "[i]t will never be an encyclopedia, but it will contain extensive knowledge that is quite valuable for different purposes", and Bill Thompson who stated "I use the Wikipedia a lot. It is a good starting point for serious research, but I would never accept something that I read there without checking."

Information Today's March 2006 article concludes on a similar theme:

The inconvenient reality is that people and their products are messy, whether produced in a top-down or bottom-up manner. Almost every source includes errors... Many non-fiction books are produced via an appallingly sloppy process... In this author’s opinion, the flap over Wikipedia was significantly overblown, but contained a silver lining: People are becoming more aware of the perils of accepting information at face value. They have learned not to consult just one source.

Dan Gillmor, a Silicon Valley commentator and author commented in October 2004 that, "I don't think anyone is saying Wikipedia is an absolute replacement for a traditional encyclopedia. But in the topics I know something about, I've found Wikipedia to be as accurate as any other source I've found."

Referencing Linus' law of open-source development, Larry Sanger who is a co-founder of Wikipedia, stated on Kuro5hin in 2001 that "Given enough eyeballs, all errors are shallow."

Likewise, technology figure Joi Ito wrote on Wikipedia's authority, "[a]lthough it depends a bit on the field, the question is whether something is more likely to be true coming from a source whose resume sounds authoritative, or a source that has been viewed by hundreds of thousands of people (with the ability to comment) and has survived."

References

See also

Articles

• Criticism of Wikipedia
• Internet encyclopedia project
• List of encyclopedias
• Wikipedia
• Wikipedia community
• Essjay controversy

Wikipedia project pages

• Wikipedia:Researching with Wikipedia
• Wikipedia:Wikipedia in academic studies (list of studies)
• Wikipedia:Wikipedia as an academic source (list of cited uses)
• Wikipedia:Press coverage
• Wikipedia:Replies to common objections
• Wikipedia:Statistics

External links

• Stanford University's Web Credibility research lab, researches similar issues on credibility measures of web information.
• Librarians' Claims and Opinions Regarding Wikipedia