PREreview del GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents

This paper studies whether an LLM judge, used with simulated users, can choose the better task-oriented agent. Across 25 agent settings, the judge ranks agents well when their performance differs greatly, but is less reliable when they perform similarly. The paper also shows that a conversation can feel satisfying even when the task fails. This moves the field forward by testing evaluation scores against task outcomes and by focusing on the agent choices teams make before release.

Major issues

• The 57.5% failure rate comes from conversations rated satisfactory by a human panel in a selected sample. On the main agent set, the full judge has a 20% failure rate among accepted cases. The paper should make this difference clearer so readers do not mistake the first number for the judge’s error rate.

• The reported 31% wrong-choice rate concerns agents with very close task results. Some of these differences may be due to measurement noise. Please show how often the task results clearly support one agent over the other.

• The study uses simulated users. It remains unclear how well the findings apply to real customer conversations.

Minor issues

• A short example near the start would help explain the difference between satisfaction-only scoring and the full judge.

• The discussion of the “free completion bit” should state earlier that it mainly catches conversations cut short, rather than most task failures.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.