Imitation Game: Are You Talking To A Human Or A Computer?

Alan Turing called it the Imitation Game. It was 1950 and the father of modern computing already knew there’d come a time when machines could imitate man, when they’d become so “clever” we wouldn’t know if we were talking to a computer or a person. That time is now.

AI21 Labs, the Israeli artificial intelligence company, has used the the latest advances to create a two-minute game version of what became known as the Turing Test. It really works and you can try it here.

Users have to decide whether they’ve been chatting with another user or with an AI bot (software program).

It’s a fun game with a serious purpose – to show how far artificial intelligence has advanced.

I didn’t think it would fool me, but it did. It even fooled Amos Meron, the guy who invented it. “I created the game and I don’t have a perfect strike rate,” he’s happy to admit to NoCamels.

He and his team did everything they could to make the AI in Human or Not? as convincing as possible, even down to the typos it deliberately makes, the rude words it uses, the pauses before it responds – as if it’s thinking – or its capacity to lose interest half-way through and abandon the chat, like a bored teenager.

ChatGPT’s stiff formality and groveling apologies were never going to cut it, so they had to teach AI to be “normal.”

“For the most part, the bots are trained to act as players in the game, which means they might turn the tables and ask you if you’re a human,” says Meron.

And so it becomes a game of double-bluff. The user pretends to be a bot to dupe the actual bot, but in the end the bot wins – because it’s constantly learning new tricks that it can use next time.

Meron and his team use a range of AI bots or LLMs – larger language models – including ChatGPT4 and AI21’s own Jurassic-2.

“They generate a character, so each bot has a name, and a place where they live, a date of birth, a background, a whole personality, knowledge of recent events, what the weather is at the moment,” he says.

“Some bots are more formal – more British if you like – others are American, Canadian, whatever. They use the slang that their character would use.

“If they’re from South America they might use a few words of Spanish, they make typos, but they’re smart typos, like a ‘G’ and an ‘H’ because they’re next to each other on the keyboard, or they might mix up ‘then’ and ‘them’.”

If his bots spoke like ChatGPT nobody would ever be fooled, he says.

“If you asked ChatGPT what year Alan Turing was born, it would tell you, Alan Turing was born in 1912, like a Wikipedia article. If you asked our bot he might say ‘f*** do I know?’ because that’s what humans might say.”

A two-minute chat is, he says, the sweet spot. Anything shorter would allow only a couple of exchanges. A longer conversation might bore the user, and may also expose the flaws in the bot. Fooling a user for two minutes is easier than half an hour.

He says it took about three months for him and his team to develop the game, which is a research project for AI21, a company headed by Amnon Shashua, the startup billionaire behind Mobileye, ONE ZERO Digital Bank and OrCam. 

“It might lead to a scientific paper down the line, but at the moment, we’re focusing on it as a social experiment,” he says. There’s no clear commercial value to it at this stage.

They’ll keep it online for a few weeks before turning their attention to other projects.

“We want to explore what AI is capable and isn’t capable of in April 2023, and I say April [and not simply 2023] because things are changing so fast.”

The Human or Not? Game has so far been played many hundreds of thousands of times. Meron is not ready, yet, to reveal the balance between when it’s fooled people and when the bots have been found out, but he says he will do at some point.

How would Alan Turing, the genius mathematician and computer scientist who famously cracked the codes used by German Enigma machines during World War II, react to Meron’s experiment?

“I have two responses, and not sure which one I’m leaning to,” he says.

“One is it that he would be amazed at like what he thought was just a theoretical experiment is actually alive and working. The other is that he would be surprised that it took us so long.

“I’m only 35 but when I was a child what we’re doing now seemed like science fiction very far in the future. We are standing on the shoulders of giants.

“Maybe back then it made total sense to him that in just a few short decades, there would be a lot of like machines that could pass the Turing Test.”

Subscribe to NoCamels weekly newsletter and get our top stories