Eugene Goostman

Eugene Goostman is a chatterbot. First developed by a group of three programmers; the Russian-born Vladimir Veselov, Ukranian-born Eugene Demchenko, and Russian-born Sergey Ulasen in Saint Petersburg in 2001, Goostman is portrayed as a 13-year old Ukranian boy—a trait that is intended to induce forgiveness in users for his grammar and level of knowledge.

The Goostman bot has competed in a number of Turing test contests since its creation, and finished second in the 2005 and 2008 Loebner Prize contest. In June 2012, at an event marking what would have been the 100th birthday of their namesake, Alan Turing, Goostman won what was promoted as the largest-ever Turing test contest, successfully convincing 29% of its judges that it was human. On 7 June 2014, at a contest marking the 60th anniversary of Turing's death, 33% of the event's judges thought that Goostman was human; the event's organizer Kevin Warwick considered it to have "passed" Turing's test as a result, per Turing's prediction that by the year 2000, machines would be capable of fooling 30% of human judges after five minutes of questioning.

The validity and relevance of Goostman's "pass" was questioned by critics, who noted the exaggeration of the "achievement" by Warwick and the event's organizers, the bot's use of personality and humour in an attempt to misdirect users from its non-human tendencies and lack of actual intelligence, along with "passes" achieved by other chatbots at similar events in the past.

Personality

Eugene Goostman is portrayed as being a 13-year-old boy from Odessa, Ukraine, who has a pet guinea pig and a father who is a gynaecologist. Veselov stated that Goostman was designed to be a "character with a believable personality". The choice of age was intentional—as, in Veselov's opinion, a thirteen-year-old is "not too old to know everything and not too young to know nothing". Goostman's young age also induces people who "converse" with him to forgive minor grammatical errors in his responses. In 2014, work was made on improving the bot's "dialog controller", allowing Goostman to output more human-like dialogue; future work will also be made on improving Goostman's "conversation logic".

Accolades

Eugene Goostman competed in a number of Turing test competitions, including the Loebner Prize contest; it finished joint second in the Loebner test in 2001, and came second to Jabberwacky in 2005 and to Elbot in 2008. On 23 June 2012, Goostman won a Turing test competition at Bletchley Park in Milton Keynes, held to mark the centenary of Alan Turing. The competition, which featured five bots, twenty-five hidden humans, and thirty judges, was considered to be the largest-ever Turing test contest by its organizers. After a series of five-minute-long text conversations, 29% of the judges were convinced that the bot was an actual human.

2014 "pass"

Kevin Warwick considered Goostman to be the first machine to "pass" a Turing test, although the validity of his claim was disputed by critics.

On 7 June 2014, in a Turing test competition at the Royal Society, organized by Kevin Warwick of the University of Reading to mark the 60th anniversary of Turing's death, Goostman won after 33% of the judges were convinced that the bot was human. 30 judges took part in the event, which included Lord Sharkey, a sponsor of Turing's posthumous pardon, and Red Dwarf actor Robert Llewellyn. Each judge simultaneously participated in five textual conversations, each of them between a human and one of the five competing bots. In all, a total of 300 conversations were conducted. In Warwick's view, this made Goostman the first machine to pass a Turing test. He added that "some will claim that the Test has already been passed. The words Turing Test have been applied to similar competitions around the world. However this event involved more simultaneous comparison tests than ever before, was independently verified and, crucially, the conversations were unrestricted. A true Turing Test does not set the questions or topics prior to the conversations."

In his 1950 paper "Computing Machinery and Intelligence", Turing predicted that by the year 2000, computer programs would be sufficiently advanced that the average interrogator would, after five minutes of questioning, "not have more than 70 per cent chance" of correctly guessing whether they were speaking to a human or a machine. Although Turing phrased this as a prediction rather than a "threshold for intelligence", commentators believe that Warwick had chosen to interpret it as meaning that if 30% of interrogators were fooled, the software had "passed the Turing test".

Reception

The validity of Warwick's claim that Eugene Goostman was the first ever chatbot to pass a Turing test was met with skepticism; critics acknowledged similar "passes" made in the past by other chatbots under the 30% criteria, including PC Therapist in 1991 (which tricked 5 of 10 judges, 50%), and at the Techniche festival in 2011, where a modified version of Cleverbot tricked 59.3% of 1334 votes (which included the 30 judges, along with an audience). Cleverbot's developer, Rollo Carpenter, argued that Turing tests can only prove that a machine can "imitate" intelligence rather than show actual intelligence. Imperial College London professor Murray Shanahan questioned the validity and scientific basis of the test, stating that it was "completely misplaced, and it devalues real AI research. It makes it seem like science fiction AI is nearly here, when in fact it’s not and it’s incredibly difficult."

Gary Marcus was critical of Warwick's claims, arguing that Goostman's "success" was only the result of a "cleverly-coded piece of software", going on to say that "it’s easy to see how an untrained judge might mistake wit for reality, but once you have an understanding of how this sort of system works, the constant misdirection and deflection becomes obvious, even irritating. The illusion, in other words, is fleeting." While acknowledging IBM's Deep Blue and Watson projects—single-purpose computer systems meant for playing chess and Jeopardy! respectively—as examples of computer systems that show a degree of intelligence in their specialized field, he further argued that they were not an equivalent to a computer system that shows "broad" intelligence, and could—for example, watch a television programme and answer questions on its content. Marcus stated that "no existing combination of hardware and software can learn completely new things at will the way a clever child can." However, he still believed that there were potential uses for technology such as that of Goostman, specifically suggesting the creation of "believable", interactive video game characters.

Criticism was also directed towards the background of Kevin Warwick himself, who is an established, but controversial figure in artificial intelligence research. Some of his peers have claimed that his work lacks scientific rigour and is more "entertainment" than academic research, such as when he had a microchip implanted in his arm to become a "cyborg". The Register, a technology news website which has historically referred to him as "Captain Cyborg", described Warwick as an "attention-seeking academic" with a history of making "improbable claims to the press", considering the event and its exaggerated media coverage to be a publicity stunt. Mike Masnick, editor of the blog Techdirt, made similar assessments in relation to Eugene's "pass", stating that Warwick had a history of making "ridiculous" statements that "gullible" media outlets would "repeat without question", as they had following his announcement of Eugene's alleged pass.

Masnick also argued that "almost everything about the story is bogus", specifically citing the exaggerated statements made in the press release (such as referring to the five chatbots as being "supercomputers" instead of merely scripts), the aforementioned "passes" by Cleverbot and PC Therapist, a belief that Goostman's developers "gamed" the rules of the test by claiming the bot was only 13 years old, that "you don't get to run a single test with judges that you picked and declare you accomplished something", and the concept of Turing tests as a whole—arguing that "creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence."

See also

• Natural language processing

External links

• Official website