PREreview del A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases

This paper addresses a critical gap in enterprise applications of large language models by introducing a benchmark tailored to question answering over enterprise SQL databases. While much prior Text-to-SQL research focuses on academic or simplified schemas, this work explicitly targets enterprise settings, where complex schemas, business metrics, and domain-specific semantics pose significant challenges for LLM-based systems.

A key contribution is the inclusion of a knowledge graph layer that captures enterprise ontologies and mappings over an insurance-domain SQL schema. This contextual layer enables the authors to systematically evaluate the impact of semantic enrichment on question answering accuracy. The empirical results are compelling: zero-shot GPT-4 achieves low accuracy when operating directly over SQL, while accuracy improves substantially when questions are posed over the knowledge graph representation. This finding provides concrete evidence that semantic context, rather than raw schema access alone, is critical for reliable enterprise question answering.

One limitation of the study is its reliance on a single domain and schema, which may constrain generalizability across industries. Additionally, the benchmark focuses primarily on accuracy, leaving open questions regarding latency, cost, and maintenance overhead associated with knowledge graph construction.

Overall, this paper makes a valuable contribution by providing both a realistic enterprise benchmark and strong empirical support for the role of knowledge graphs in improving LLM-based analytics, with clear implications for governed BI and semantic-layer design.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.