Loebner Prize: A Turing Test Challenge

Author

Reads 8.7K

Trophies and Medals on a White Table
Credit: pexels.com, Trophies and Medals on a White Table

The Loebner Prize is a Turing Test challenge that has been running for decades, and it's still going strong. The first Loebner Prize was awarded in 1990.

The Loebner Prize is named after its founder, Hugh Loebner, who was a computer enthusiast and a pioneer in the field of artificial intelligence. He wanted to create a competition that would push the boundaries of AI development.

The Turing Test is a measure of a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human. The Loebner Prize challenges AI developers to create a program that can pass this test.

In the Loebner Prize competition, human evaluators engage in natural language conversations with both human and machine participants, without knowing which is which. The goal is to see if the evaluators can tell the difference between the two.

Here's an interesting read: Adt Comp Sci

What Is the Prize?

The Loebner Prize is an international contest run by the AISB where chatbots compete with each other to find the most humanlike.

Credit: youtube.com, Ask a ChatBot: What is the Loebner Prize for AI? | Head Squeeze

The prize is incredibly challenging, as recreating human conversation is a complex task that covers a vast multitude of topics.

No internet access is allowed in the contest, which is done to avoid the possibility of cheating by having a human operator pretending to be the bots and typing their responses.

Entering the Prize

To enter the Loebner Prize, any chatbot from around the world is welcome to compete. The chatbot must follow certain rules available on the AISB website.

The main rules being that the bots have to use a special protocol to talk to the judges and the bot must work as a standalone piece of software on a local network.

Once the deadline for entries has passed, there is a qualifying round to determine the 4 best chatbots. This is done by asking each of the bots the same 20 questions and scoring them on the humanlike nature of their answers.

The questions vary from a simple, “How old are you?” to the much more complex, “When might I need to know how many times a wheel has rotated?”, which even I would struggle with!

Being humanlike is not the same as being intelligent though.

The Main Event

Credit: youtube.com, Ask a ChatBot: Head Squeeze chats to 2013 Loebner Prize Winner Mitsuku | Head Squeeze

It's just before 1:30pm, and the Loebner Prize contest is about to begin. All bots, judges, and human confederates are in place and ready for action.

The contest takes place over 4 rounds, with each judge talking to each of the humans and programs. The first round is underway, and everything is working fine - messages are coming back and forth between the bots and judges.

The judge I had this time round was the classic Loebner Prize judge, who was interrogating and testing Mitsuku with questions like "What is bigger, a boat or a car?" and "What's the next number in the sequence 1 2 4 8".

Bletchley Park

Bletchley Park is the venue for the Loebner Prize Final. I arrived at Bletchley Park just after 10am.

There are at least two rooms used in the competition, one for the computer programs and human confederates, and another for the judges and public viewing area. It's essential to keep the judges and confederates separated.

Credit: youtube.com, ShmooCon 2014: History of Bletchley Park and How They Invented Cryptography and the Computer Age

I always like to find the computer where I need to install Mitsuku, just to make sure everything is okay. Will Rayer, the developer of Uberbot, was already on site configuring his entry.

Each device is a Windows 7 laptop, so I installed Mitsuku in a few minutes and made sure the software loaded before waiting for the rest of the competition network to be set up.

Round 1

The Main Event was about to kick off, and I was in the room with the computers, eagerly waiting for the contest to begin. All the bots, judges, and human confederates were in place and ready for action.

At 1:30 pm, a message appeared on our consoles saying "new round", and the contest was underway. The Loebner Prize contest takes place over 4 rounds, with each judge interacting with each of the humans and programs.

The first round was nerve-wracking, as we'd tested the bots earlier but there was still a chance that something might go wrong. Fortunately, all worked smoothly, and a message saying "hi how are you today?" appeared on Mitsuku's console.

An artist’s illustration of artificial intelligence (AI). This illustration depicts language models which generate text. It was created by Wes Cockx as part of the Visualising AI project l...
Credit: pexels.com, An artist’s illustration of artificial intelligence (AI). This illustration depicts language models which generate text. It was created by Wes Cockx as part of the Visualising AI project l...

Mitsuku responded well, sending a message back saying "Ah. Pretty good thanks How about you?". I was reasonably pleased with her performance, although she could have been better in the early part of the round.

The judge's tactic with Mitsuku was to have a general conversation, and she responded okay but could have been better in the early part of the round. I was checking Tutor's computer at the same time, and saw an interaction between the judge and Tutor.

The first judge's conversation with Mitsuku was a general one, and she responded okay but could have been better in the early part of the round. This was just the beginning of the contest, and I was eager to see how the rest of the rounds would go.

Round 3

Round 3 was a bit of a letdown, with Mitsuku, Tutor, and Colombina performing well, but Uberbot missing out altogether due to a miscommunication that left it waiting for a new round signal.

An artist’s illustration of artificial intelligence (AI). This image represents how machine learning is inspired by neuroscience and the human brain. It was created by Novoto Studio as par...
Credit: pexels.com, An artist’s illustration of artificial intelligence (AI). This image represents how machine learning is inspired by neuroscience and the human brain. It was created by Novoto Studio as par...

Uberbot's problem was caused by a miscommunication that led it to think the new round signal hadn't been sent yet, and by the time it realized its mistake, it was too late to participate.

Mitsuku's judge in this round was particularly tough, asking a lot of contextual questions and even intentionally misspelling things to try and catch the bot out.

Mitsuku handled this line of questioning surprisingly well, responding okay to most of the judge's attempts to trip it up.

In this round, the importance of clear communication between the chatbot and the system became starkly apparent, as Uberbot's failure to receive the new round signal had disastrous consequences.

Other Nominees and Winners

Artemis was a nominee in 2008, but unfortunately, I don't have more information about this particular event.

In 2012, two notable nominees were The Professor and Adam, both of whom likely brought their unique personalities to the table.

Mitsuku took home the prize in 2013, making it a notable year for this winning personality.

Chip Vivant was a nominee in 2009, showing that this individual had a strong presence in the competition.

Here's a list of some of the other notable nominees and winners:

  • Artemis (2008)
  • The Professor (2012)
  • Mitsuku (2013)
  • Chip Vivant (2009)
  • Adam (2012)

Results!

Credit: youtube.com, Loebner Prize for Artificial Intelligence 2009

The results of the Loebner Prize are always a thrilling moment. The scoring system had changed this year, with each judge ranking the 4 bots in order from best to worst and giving them a score.

The scoring system used a simple points system: Best — 5 points, 2nd — 3 points, 3rd — 2 points, and 4th — 1 point. This system allowed the judges to clearly express their opinions on the bots' performances.

The scores from the four judges were added together and a percentage was calculated. I'm not sure how the percentage was arrived at, but I assume it was a percentage of the maximum score, which was 20.

The results of the contest were announced by Bertie, who explained that no bot had managed to fool any of the judges. This was to be expected, as 25 minutes to try and fool someone is incredibly difficult.

Credit: youtube.com, Loebner Prize on University Challenge

The award for the most humanlike program went to the bot with the highest percentage score. The top 4 bots and their scores were: Mitsuku — 33%, Tutor — 30%, Colombina — 25%, and Uberbot — 23%.

Here are the results in a table:

I was overjoyed to see that I had won the Loebner Prize for a 4th time, joining an exclusive club of only 2 other people who had achieved this feat.

Jay Matsuda

Lead Writer

Jay Matsuda is an accomplished writer and blogger who has been sharing his insights and experiences with readers for over a decade. He has a talent for crafting engaging content that resonates with audiences, whether he's writing about travel, food, or personal growth. With a deep passion for exploring new places and meeting new people, Jay brings a unique perspective to everything he writes.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.