Beyond the
Turing Test:

Evaluating AI Agents in Group Conversation

Hoshen Lugasi¹ Uri Berger^1,2 Niv Eckhaus¹ Itay Itzhak^1,3 Gabriel Stanovsky¹

¹ School of Computer Science and Engineering, The Hebrew University of Jerusalem

² School of Computing and Information Systems, University of Melbourne

³ Technion - Israel Institute of Technology

Flow of the Social Imitation Game. Discussion and voting continue until the AI agent is eliminated or only the AI agent and one human player remain. — **Flow of the Social Imitation Game.** Discussion and voting continue until the AI agent is eliminated or only the AI agent and one human player remain.

Abstract

Turing's Imitation Game tests machine intelligence through one-on-one turn-based conversation, but it does not capture the social dynamics of free-form human communication in group settings, where participation patterns and adherence to group norms complement linguistic content. To address this limitation, we introduce the Social Imitation Game, a communication game in which a group of human participants collectively attempts to identify which group member is an AI agent through successive rounds of conversation, each followed by a vote on the AI's identity. This design produces a natural gradient of difficulty and score, as the most suspected player is eliminated after each round.

We develop an LLM-based agent comprising a foreground pipeline that decides when to send a message and what message to generate, along with background-worker subagents that perform longer tasks, involving reasoning over the conversation state, retrieving external information, and maintaining memory. This architecture enables the agent to remain responsive while drawing on information accumulated across messages and rounds to support coherent social behavior.

We evaluate the agent by collecting a novel dataset of Social Imitation Games involving hundreds of unique human players. The results show that our agent performs almost on par with human participants in terms of elimination and votes received, leaving little room for further improvement to achieve human parity. Further semantic analysis of the messages suggests that the agent takes a more reactive role in conversations, asking fewer questions than human participants. Together, our game formulation, agent architecture, and dataset provide a new framework for studying socially situated language agents beyond the classic Imitation Game.

Our code and data is publicly available at https://hoshenlugasi.github.io/Beyond-the-Turing-Test/.

The Social Imitation Game

The Social Imitation Game places one AI agent among human participants in a group chat. The human players know that exactly one participant is an AI, but do not know which one, and their goal is to identify and eliminate it.

Game progress: interleaved discussion and voting rounds.

A game involves several players, including a group of N human participants and one AI agent. Each player's identity is hidden behind a game-assigned pseudonym. The game proceeds in timed discussion rounds, where players are free to discuss any topic, and every player can send messages to the group chat at any time.

At the end of the predefined time, the discussion round ends and a voting round takes place. Each player casts one anonymous vote for another remaining player, indicating that they believe this player is the AI agent. The player who receives the most votes is then eliminated from the game and their identity (human or AI) is revealed.

In case of a tie, a player is chosen at random between those who received most votes.

The remaining surviving players begin another discussion round without the eliminated player.

Game conclusion: AI elimination or survival.

The game concludes under one of two conditions. First, the game may end when the AI is eliminated in a voting round, at which point its identity as an AI agent is revealed to the group.

Second, the game can end when the AI survives N-1 rounds and is left playing against a single human counterpart. In this scenario, the game concludes after the (N-1)th voting round, informing the participants that they have failed to identify the AI agent, and it has survived through the game.

Evaluation and tested skills.

The game also provides a natural performance gradient. With more players, the AI agent may find it easier to blend in or avoid suspicion by limiting its participation. As the game progresses and fewer players remain, however, the group can interrogate each player more thoroughly, making it harder for the AI agent to survive.

Agent Architecture

A three-stage foreground pipeline—Scheduler, Generator, and Reviewer—decides when the agent should speak and which message to send. In parallel, specialized asynchronous background workers analyze the conversation and write their latest results to a shared file. Each foreground poll reads the latest completed contents without waiting for unfinished background analysis.

Three-Stage Foreground Pipeline

Step I: Scheduler.

Based on the current conversation poll and the shared file to which the subagents write their results, the Scheduler makes a binary SEND or WAIT decision. A WAIT decision ends the current iteration, and the loop begins again with a new poll. A SEND decision instead continues the current iteration to the remaining steps.

Step II: Generator.

The Generator produces k candidate messages suited to the current group conversation and the information in the shared file. Each candidate must express a distinct purpose rather than paraphrase another candidate.

The Generator may also abstain if it cannot identify a suitable message, which ends the current iteration.

Step III: Reviewer.

The Reviewer decides which Generator candidate, if any, should be posted to the group conversation.

Asynchronous Background Workers

While the foreground pipeline focuses on responding to the changing conversation, asynchronous background workers provide additional time for longer-running processing. Each worker comprises a goal, the logic used to achieve it through LLM calls or deterministic code, and a specified frequency at which it should be run. This modular framework supports the addition of specialized processes without reducing the agent's responsiveness.

To enable the background workers to communicate with the foreground processing in an asynchronous manner, we introduce a shared filesystem. Each worker writes the outcome of its latest run to the filesystem, allowing the foreground pipeline to incorporate those results at its different steps.

Before posting a chosen message, the agent waits for a message-length-dependent delay that simulates human typing, with the elapsed foreground computation time counting toward that delay. It then polls the group chat again. The message is suppressed if another participant has already made the same contribution, the discussion has moved on, or a newer message makes it inappropriate. Otherwise, the selected candidate is posted to the group chat, and the foreground loop begins again.

Results

Survival during live games

The agent wins (i.e., it survives all rounds) in 15 of 50 games (30.0%), compared with an expected rate of 35.5% under chance-level elimination. The difference is -5.5 percentage points, with a 95% bootstrap confidence interval from -17.3 to +7.2 percentage points.

In the first round, 22.5% of human votes target the agent, close to the 21.4% expected under random guessing.

We focus on the first round, as analysis of later rounds is biased, since they occur only when the agent survives earlier votes and end when it is eliminated.

The agent resembles humans in participation rate but differs in conversational role.

The agent sends the same average number of messages per round as humans (3.84), and its messages are similar in length (4.44 versus 4.59 words).

We find that the agent answers questions more often than humans but asks fewer questions, indicating a more reactive conversational role.

Participants rate the agent as human-like, but their comments mention a few recurring signs of AI behavior.

The mean post-game human-likeness rating is 3.93/5. Participants rate message relevance slightly higher and their timing slightly lower than that.

In their free-text responses, they often mention the agent produces some generic answers, too little personal detail, and unusually clean language.

Other participants say that nothing in particular seemed artificial.

Code

Data Collection

We collect a dataset of Social Imitation Game games to evaluate our agent and support future study of agents in group settings.

The dataset contains 50 games with 234 distinct human participants, 135 discussion rounds, and 2,414 public messages.

Each participant is allowed to play once. Thirty-four games include five humans and the agent, and sixteen include four humans and the agent. Each discussion round lasts 2.5 minutes.

We record all messages and timestamps, active players, votes and eliminations.

Post-game survey results

Of the 234 human participants, 229 submitted a post-game survey. Human-likeness, timing and message pacing, and relevance and coherence were rated in both survey versions and therefore have 229 responses. The enjoyment item was included in the web survey and has 224 responses.

Do not attempt re-identification.

DATASET.md Dataset

Conclusion and Ethical Considerations

While our goal in this work was to assess the capabilities of LLMs in group conversations, our findings raise the risk that natural human-looking participation in an online conversation may be mistaken for evidence that a participant is human.

Our results therefore show only that our system achieves human-like identification outcomes under this protocol, not that its architecture is optimal.

BibTeX

@unpublished{lugasi2026beyond,
  title  = {Beyond the {Turing} Test: Evaluating {AI} Agents
            in Group Conversation},
  author = {Lugasi, Hoshen and Berger, Uri and Eckhaus, Niv
            and Itzhak, Itay and Stanovsky, Gabriel},
  year   = {2026},
  note   = {Unpublished manuscript},
  url    = {https://hoshenlugasi.github.io/Beyond-the-Turing-Test/}
}

CITATION.cff

Beyond theTuring Test: