PREreview of AI Models Excel at Orchestration but Falter at Biological Judgment: Findings from an Agentic Gene Annotation Study

Distinguishing Decision Coverage, Adjudication Accuracy, and Orchestration Efficiency in Agentic Gene Annotation

Review of: Sinha, A. AI Models Excel at Orchestration but Falter at Biological Judgment: Findings from an Agentic Gene Annotation Study. bioRxiv (2026). https://doi.org/10.64898/2026.09.23.753765

Overall assessment

This preprint addresses an increasingly important question in scientific AI: whether competence in coordinating analytical workflows should be taken as evidence of competence in interpreting their biological outputs. By separating deterministic sequence analysis, LLM-guided evidence acquisition, and final adjudication, Genome Skeptic offers a useful experimental framework for examining these capabilities independently.

The prospectively locked comparisons, explicit scoring rules, and publicly available computational artifacts are notable strengths. Independent recalculation from the available case-level records reproduced the principal paired results, including the decrease from 18/20 to 11/20 correct classifications (exact two-sided McNemar P = 0.015625). I found no numerical inconsistency in this comparison.

My principal concerns therefore concern interpretation rather than numerical correctness. In particular, the findings invite a more explicit distinction between decision coverage, correctness conditional on making a decision, and the incremental value of LLM-guided evidence acquisition. Clarifying these dimensions would strengthen the study's implications for scientific agent design.

Major comment 1. Distinguishing abstention from incorrect biological adjudication

The reduction in overall accuracy when Sol assumes final decision authority is statistically supported. However, its mechanism deserves greater emphasis.

Independent examination of the locked case-level outputs confirms that all nine disagreements between the deterministic and LLM adjudicators involve transitions from ABSENT to UNRESOLVED, rather than incorrect definitive positive calls. Under a secondary selective-classification interpretation that treats UNRESOLVED as abstention, the LLM provides definitive classifications for 11/20 cases, all of which are correct. Thus, decision coverage is 55%, whereas accuracy conditional on a definitive classification is 100%.

These secondary figures do not replace the prespecified accuracy endpoint. The original rule appropriately penalizes unresolved outputs when the task requires definitive classifications against resolved reference labels. Nevertheless, the distinction matters: a system that produces more incorrect definitive classifications poses a different problem from one that withholds conclusions despite sufficient evidence.

The observed reduction in specificity from 1.00 to 0.30 likewise reflects the handling of unresolved negative cases, rather than an increase in false-positive assertions.

I suggest reporting coverage and conditional accuracy alongside the prespecified overall accuracy, ideally accompanied by the complete transition matrix and class-specific abstention rates. This would preserve the main finding while identifying its operational mechanism more precisely.

Major comment 2. What establishes the value of LLM-guided orchestration?

The reduction in follow-up analyses without a corresponding change in observed endpoint accuracy is an important result. In the 20-case comparison, the Sol-controlled strategy required 39 follow-up analyses, compared with 57 under the fixed strategy and 74 under the exhaustive strategy.

However, the unchanged endpoints across controller strategies also raise a question about how sensitive the task is to evidence-acquisition decisions.

If different policies obtain different intermediate evidence yet consistently produce identical final classifications, the observed reduction in analysis calls establishes operational economy, but does not by itself establish that the LLM selects analyses more effectively than a simpler policy.

A budget-matched, non-LLM baseline—such as a fixed-priority heuristic, greedy selection rule, or random policy—would help distinguish the value of adaptive LLM routing from the value of avoiding analyses that rarely alter final decisions.

Such a comparison need not diminish the demonstrated reduction in analysis calls. Rather, it would clarify which part of the observed efficiency gain is attributable specifically to LLM-guided selection. If additional experiments are impractical, explicitly limiting the claim to fewer follow-up actions with unchanged observed endpoints would be sufficient.

Major comment 3. Interpreting the prospective benchmark and its generalizability

The 60-case benchmark provides a valuable additional perspective, but its relationship to the 20-case role-swap experiment should remain explicit.

The benchmark used Qwen3:4b in its primary agentic arm, whereas the 20-case role-swap experiment evaluated Sol. These results therefore address related architectural questions under different model configurations, rather than constituting a direct replication of the same model's behavior.

Among the 60 benchmark cases, 41 were classified as truth-evaluable, and the deterministic, agentic, and exhaustive Genome Skeptic strategies each achieved 33/41 correct classifications. The remaining 19 cases were classified as TRUTH_UNCERTAIN.

A further observation warrants particular attention: sensitivity for the seven truth-positive tetA cases was 0/7 across the Genome Skeptic strategies. Although the available post-hoc analysis suggests a family-state validation mechanism, this proposed explanation should remain distinct from the prospectively established result.

These observations would be easier to interpret if the results explicitly separated model configurations, evaluable denominators, uncertain-reference cases, and gene-family-specific performance. The reference-label construction also deserves careful qualification when interpreting absolute accuracy, given its overlap with components of the deterministic validation framework.

Overall recommendation

The study makes a useful contribution by experimentally separating workflow coordination from biological adjudication. Its principal numerical findings are reproducible from the examined computational records, and the observed differences between agent roles are meaningful within the evaluated task.

The suggested revisions are primarily interpretive rather than corrective. Separating abstention from incorrect definitive decisions, distinguishing reduced analysis calls from the added value of LLM routing, and clarifying the boundaries of the prospective benchmark would make the findings more informative beyond this particular annotation system.

More broadly, evaluations of scientific AI agents should distinguish three questions: How efficiently does the system acquire evidence? How often does it reach a definitive decision? And how reliable are the decisions it actually makes?

These are related capabilities, but they should not be treated as interchangeable measures of scientific competence.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.