DMOZ

The neutrality of this page is disputed.

The Open Directory Project (ODP), also known as DMoz (for Directory.Mozilla), is a multilingual open content directory of World Wide Web links owned by AOL Time Warner that is constructed and maintained by a community of volunteer editors.

History

Motivation and Founders

ODP was founded as Gnuhoo by Rich Skrenta and Bob Truel in 1998. At the time, Skrenta and Truel were working as engineers for Sun Microsystems. Chris Tolles, who worked at Sun Microsystems as the head of marketing for network security products, also signed on in 1998 as a co-founder of Gnuhoo along with co-founders Bryn Dole and Jeremy Wenokur. Skrenta was already well known for his role in developing TASS, an ancestor of TIN, the popular threaded Usenet newsreader for Unix systems. Coincidentally, the original category structure of the Gnuhoo directory was based loosely on the structure of Usenet newsgroups then in existence.

Gnuhoo to Newhoo to the Open Directory Project

The Gnuhoo directory went live on June 5, 1998, and was renamed Newhoo after a Slashdot article was posted wherein Gnuhoo's detractors claimed that Gnuhoo had nothing in common with the spirit of free software for which the GNU project was known and was simply a commercial enterprise seeking to construct an alternative to Yahoo! using volunteer labor.[1] Newhoo became ODP after it was acquired by Netscape Communications Corporation for the sum of $1 million in October of 1998 and the content was released under an open content license. Netscape was acquired by AOL shortly thereafter, and ODP was one of the assets included in the acquisition. AOL later merged with Time-Warner.

Directory Growth and Maturation

By the time Netscape assumed stewardship, the Open Directory Project had about 100,000 URLs indexed with contributions from about 4500 editors. On October 5, 1999, the number of URLs indexed by ODP reached one million. According to an unofficial estimate, the number of URLs in the Open Directory surpassed the number of URLs in the Yahoo! directory in April 2000 with about 1.6 million URLs. ODP achieved the milestone of indexing two million URLs on August 14, 2000, and the milestone of three million listings was reached on November 18, 2001. As of July 2003, ODP has about 3.8 million listings organized into over 460,000 categories and claimed to be derived from the contributions of some 57,000 editors. At that time, the number of active editor accounts (i.e. those editor logins which had not been removed, voluntarily resigned, or timed out due to inactivity of four months) was over 9,000.

Competing and Spinoff Projects

ODP has inspired the formation of at least two other Web directories edited by volunteers and sponsored by public companies: The now defunct Go directory (formerly owned by The Walt Disney Company), and Zeal (acquired by LookSmart). However, neither of these Web directories have licensed their content for open content distribution, a strategy which ensured ODP's success in a highly competitive market.

ODP has inspired the formation of a number of proprietary Web directories, some of which are edited by volunteers, such as Zeal and the multilingual MavicaNet, and others which are edited by independent contractors, such as Hotrate. Moreover, the concept of using a large-scale community of editors to compile online content has been successfully applied to other types of projects such as Wikipedia, a freely licensed online encyclopedia.

Three open content volunteer projects have been inspired by ODP's editing model: an open content restaurant directory known as ChefMoz (launched by ODP management), an open content music directory known as MusicMoz, and an encyclopedia known as Open Site. However, none of the three have yet to achieve success at the level of ODP.

ODP Content

Organization and Scope of Content

ODP uses a hierarchical ontology scheme for organizing site listings. Listings on a similar topic are grouped into categories, which can then include smaller categories.

Gnuhoo borrowed its initial ontology from Usenet. For example, the topic covered by the comp.ai.alife newsgroup was represented by the category Computers/AI/Artificial_Life. The original divisions were for Adult, Arts, Business, Computers, Games, Health, Home, News, Recreation, Reference, Regional, Science, Shopping, Society, and Sports. While these fifteen top-level categories have remained intact, the ontology of second- and lower-level categories has undergone a gradual evolution; significant changes are initiated by discussion among editors, and then implemented when consensus has been reached.

In July 1998, the directory became multilingual with the addition of the World top-level category. The remainder of the directory for English language sites only. By July 2003, sixty-seven languages were represented. While the English component of the directory held almost 75% of the sites, the growth rate of the non-English components of the directory was greater than the English component through 2002 and 2003. Ontology in non-English categories generally mirrors that of the English directory, although exceptions which reflect language differences are quite common.

Several of the top-level categories have unique characteristics. The Adult category is not present on the directory homepage, but it is fully available in the RDF dump that ODP provides. While the bulk of the directory is categorized primarily by topic, the Regional category is categorized primarily by region. This has led many to view ODP as two parallel directories: Regional and Topical.

On November 14, 2000, a special directory within the Open Directory was created for people under 18 years of age. Key factors distinguishing this "Kids and Teens" [2] area from the main directory are:

Stricter guidelines which limit the listing of sites to those which are targeted or appropriate for people under 18 years of age.[3]
Category names as well as site descriptions use vocabulary which is age appropriate.
Age tags on each listing distinguish content appropriate for kids (age 12 and under), teens (13 to 15 years old) and mature teens (16 to 18 years old).
Kids and Teens content is available as a separate RDF dump.
Editing permissions are such that the community is parallel to that of the Open Directory.

By July 2003, this portion of the Open Directory included over 22,000 site listings.

Directory Maintenance

Directory listings are maintained by editors on a daily basis. While some editors focus on the addition of new listings, others focus on maintaining the existing listings. This includes tasks such as the editing of individual listings to correcting spelling and/or grammar errors, as well as monitoring the status of sites which are linked to.

Robozilla is a web crawler written to check the status of all sites listed in ODP. Periodically, Robozilla will flag sites which appear to have moved or disappeared, and editors follow up to check the sites and take action. This process is critical for the directory in striving to achieve one of its founding goals: to reduce the link rot in web directories.

Another challenge for ODP is how to deal with site listings which point to domains which have changed ownership and no longer offer the content upon which the listing was based (i.e. domain hijacking). Due to the popularity of the Open Directory and its resulting impact on search engine rankings, listed domains which have lapsed registration are particularly attractive. Editors consistently attempt to locate and deal appropriately with these listed domains.

License and Requirements

ODP data is made available for open content distribution under the terms of the Open Directory License which requires a specific ODP attribution table on virtually every Web page that uses the data. However, not all data is included: a large number of categories, some "Test" categories, and private editor notes and forums are excluded. Furthermore, the RDF dumps have gained a reputation in some circles for being frequently corrupted.

Users of ODP content

At the height of ODP's popularity, ODP data powered the core directory services for many of the Web's largest search engines and portals, including Netscape Search, AOL Search, Google, Lycos, HotBot, and DirectHit. However, most of these search engines have stopped updating their ODP data, and some smaller sites stopped using RDF dumps, as they grew increasingly large, choosing to query live data directly from the ODP website. Moreover, many noteworthy Web portals that once embraced ODP's free data co-branding have since abandoned ODP's free data model for the revenue sharing model of ODP's major competitor LookSmart.

ODP Policies and Procedures

Becoming an editor

There are restrictions imposed on who can become an ODP editor. The primary gatekeeping mechanism is an editor application process, presided over by ODP's meta editors, wherein editor candidates are required to demonstrate their editing abilities and disclose any and all website affiliations that might pose a conflict of interest. Approximately 90% of these applications are rejected, but re-application is the norm.

Editing model

ODP's editing model is a hierarchical one. Upon becoming an editor, an individual will generally have editing permissions in only a small category. Once they have demonstrated basic editing skill in compliance with the Editing Guidelines, they are welcome to apply for additional editing privileges, in either a broader category, or in a category elsewhere in the directory. Mentorship relationships between editors are encouraged, and internal forums provide a vehicle for new editors to ask questions.

Over time, senior editors may be granted additional privileges which reflect their editing experience and leadership within the editing community. The most straightforward are editall privileges, which allow an editor to access all categories in the directory. Meta privileges additionally allow editors to perform tasks such as reviewing editor applications, setting category features, and handling external and internal abuse reports. Cateditall privileges are similar to editall, but only for a single directory category; similarly, catmod privileges are similar to meta, but only for a single directory category. Catmv privileges allow editors to make changes to directory ontology by moving or renaming categories. All of these privileges are granted by staff, usually after discussion with meta editors.

Editing Guidelines

All ODP editors are expected to abide by ODP's Editing Guidelines.[4] These guidelines describe: what types of sites may be listed and which may not; how site listings should be titled and described in a loosely consistent manner; conventions for the naming and building of categories; conflict of interest limitations on the editing of sites which the editor may own or otherwise be affiliated with; and a code of conduct within the community. Editors who are found to have violated these guidelines may be contacted by staff or senior editors, have their editing permissions cut back, or, as a last resort, lose their editing privileges entirely. ODP Guidelines are periodically revised after discussion in editor forums.

Site submissions

The original motivation for forming Gnuhoo/Newhoo/ODP was the frustration that many people experienced in getting their sites listed on Yahoo!. However, Yahoo! has since addressed this problem by implementing a paid service for timely consideration of site submissions, making free site submissions the primary advantage of ODP. In striking contrast, ODP now has approximately one million unreviewed site submissions, in large part due to spam and incorrectly submitted sites, making the average processing time for a site properly submitted to ODP approximately six months. Moreover, because of concerns about abusive e-mail, ODP's volunteer editors are discouraged from communicating with site submitters, leaving many submitters to wonder whether and when their site has been considered and rejected for inclusion in ODP. In response to ongoing criticism of this policy, ODP editors have set up a public forum where queries about site submission status can be posted.[5]

Controversy and Criticism

Allegations of abusive editing practices

There have long been allegations that volunteer ODP editors give favorable treatment to their own websites while concomitantly thwarting the good faith efforts of their competition. Such allegations are fielded by ODP's staff and meta editors, who have the authority to take disciplinary action against volunteer editors who are suspected of engaging in abusive editing practices. In 2003, ODP introduced a new Public Abuse Report System that allows members of the general public to report and track allegations of abusive editor conduct using an online form. [6]

In a widely publicized federal lawsuit which is still ongoing in the United States, a prominent tax law firm known as J.K. Harris obtained a temporary restraining order and then a preliminary injunction against a volunteer ODP editor. The plaintiff alleged, in part, that the editor in question had engaged in abusive editing practices that violated both state and federal laws restricting unfair competition. The Honorable Claudia Wilken, United States District Judge for the Northern District of California, found that this allegation did not justify injunctive relief, but granted injunctive relief on other grounds.[7]

Allegations of unfair competition at ODP and unethical quid pro quo also arose when ODP's paid staff gave the paid employees of professional content providers such as AOL and Rolling Stone magazine high level editing access at ODP. Many volunteer editors perceived this to be a sellout of the grass roots principles on which ODP was purportedly based.

Ownership and Management of ODP

Underlying much of the controversy surrounding ODP is its ownership and management. Many of the original GnuHoo volunteers felt that they had been deceived into joining a commercial enterprise, but most of the initial controversy died down when the project was renamed NewHoo. Moreover, when Netscape acquired the project, renamed it ODP, and released ODP's content under an open content license, ODP's critics all but disappeared. However, as the community of editors at ODP began to grow, and ODP's content became widely used by most major search engines and Web directories, the issue of ODP's ownership and management resurfaced.

At ODP's inception, there was little thought given to the idea of how ODP should be managed, and there were no official forums, guidelines, or FAQs. In essence, ODP began as a free for all. Even after ODP set up its internal editor forums, many editors remained blissfully unaware that these forums existed until they were directed to the forums by one of their fellow editors. Moreover, given that ODP had no official guidelines at first, ODP editors simply hashed out some sort of consensus among themselves and published unofficial FAQs.

As time went on, the ODP Editor Forums became the de facto ODP parliament, and when one of ODP's staff members would post an opinion in the forums, it would be deferred to as an official ruling. (I.e., "Staff has spoken.") There was also a short-lived attempt at moderation of the ODP Editor Forums, but it was abandoned as being the antithesis of the egalitarian principles on which the ODP community was supposed to be based. Even so, ODP staff began to give trusted senior editors additional editing privileges, including the ability to approve new editor applications, which eventually led to a stratified hierarchy of duties and privileges among ODP editors, with ODP's paid staff having the final say regarding ODP's policies and procedures.

Allegations that ODP editors are removed for criticizing ODP's policies

ODP's paid staff has imposed controversial policies from time to time, and volunteer editors who openly dissent often find their editing privileges removed, an ongoing situation which has been chronicled at the XODP Yahoo! eGroup since May of 2000. The first noteworthy expose was Life After the Open Directory Project, a June 1, 2000 guest column written for Traffick.com by David F. Prenatt, Jr. (former ODP editor 'netesq'), who founded the XODP Yahoo! eGroup after losing his ODP editing privileges.[8]

Another noteworthy example was the volunteer editor known by the alias The Cunctator, who was banned from the ODP soon after submitting an article to Slashdot on October 24, 2000, which criticized changes in ODP's copyright policies.[9]

In light of the apparent risks of expressing dissent openly, at least one ODP insider has expressed his or her dissent in AOL Meddling in ODP Causes Shift in Balance of Editorial Power, an article published at Traffick.com on September 4, 2001 under the pseudonym of Julian McCreary.[10]

Editor Removal Procedures

ODP's editor removal procedures, which are overseen by ODP's staff and meta editors, are very controversial. According to ODP's official editorial guidelines, editors are removed for abusive editing practices. However, discussions that may result in disciplinary action against volunteer editors take place in a private forum which can only be accessed by ODP's staff and meta editors, and volunteer editors who are at risk of losing their editing privileges may not be given any notice that such proceedings are taking place, much less notice of an adverse decision. The rationale that is publicly asserted for this policy is that volunteer editors are assumed to know when they are violating ODP guidelines.[11]

ODP has a standing policy that prohibits any current ODP editors in a position to know anything from discussing the reasons for specific editor removals. In the past, this has led to claims that many ODP editors are left to wonder why they cannot login at ODP to perform their editing work. However, ODP is now set up in such a way that when someone attempts to login at ODP using a deactivated editor login, a generic web page is displayed that informs a removed editor that a final decision has been made regarding the deactivation of his or her login and providing a list of possible reasons as to why such a decision might have been made. At the same time, a software glitch can result in the same page being displayed when an editor login has simply timed out and the editor in question is in fact eligible for reinstatement. Consequently, the ambiguity typically associated with editor removal procedures remains an ongoing issue.

Size of the directory and number of editors

As of July 2003, ODP claimed to have over 3.8 million entries in 460,000 categories, maintained by 57,238 editors. However, the number of editors contributing to ODP is exaggerated by a ratio of at least 5 to 1. This is due to the fact that ODP tracks the total number of editor logins ever created rather than the number of currently active editors. After an inactive period of three months, many of these logins time out. Other logins that are included in the overall tally represent the logins of former editors who have had their editing privileges removed, either for abusive editing practices or by consensus of ODP's staff and meta editors. Moreover, when ODP editor logins are intentionally deactivated, many former editors simply reapply under an assumed identity, leading to even greater exagerration in the number of active ODP editors.

Private editor forums

Critics of ODP have questioned the propriety of ODP having private editor forums in what is purportedly an open project, as both Go and Zeal have always allowed public access to their forum discussions. However, the defenders of ODP's private forums assert that such forums are necessary for the discussion of matters that are internal to ODP. Even so, some of these private forum discussions have been published outside of ODP after being forwarded to ODP's critics by anonymous ODP insiders. [12]

Software

Critics of ODP, most notable among them the aforementioned GNU project, also point to the fact that ODP's editing software is not free software and that ODP's content license restricts the freedom of licensees unnecessarily. As such, there have been many efforts to provide truly open alternatives to ODP (see below). These alternatives would allow communities of like-minded editors to set up and maintain their own open source/open content Web directories. However, no noteworthy open source/open content alternative to ODP has yet emerged.

References

[1] - Slashdot | The GnuHoo BooBoo | Posted by CmdrTaco (Tuesday June 23, 1998)
[2] - Open Directory Project - Kids and Teens Directory
[3] - Open Directory Project - Kids and Teens Directory Editing Guidelines
[4] - Open Directory Project Editing Guidelines
[5] - Open Directory Project Public Forum
[6] - Open Directory Project Public Abuse Report System
[7] - EFF | J.K. Harris v. Steven Kassel. (Preliminary Injunction [PDF].)
[8] - Traffick.com | Life After the Open Directory Project | Guest Column by David F. Prenatt, Jr. (June 1, 2000)
[9] - Slashdot | Dmoz (aka AOL) Changing Guidelines In Sketchy Way | Posted by CmdrTaco (Tuesday October 24, 2000)
[10] - Traffick.com | AOL Meddling in ODP Causes Shift in Balance of Editorial Power | By Julian McCreary (September 4, 2001)
[11] - DMOZ Newsletter | Editor Removal Explained by Arlarson (September 2000)
[12] - Donotgo.com | Dumb-oz (Reproduction of Internal ODP Editor Forum)