Semantic Thermodynamics of Transformer Architectures: A Framework for Understanding Hallucination Constraints
- Publicada
- Servidor
- Preprints.org
- DOI
- 10.20944/preprints202602.1962.v1
We develop \emph{Semantic Thermodynamics}, an information-theoretic framework for analyzing hallucinations in transformer systems under finite resources. The central object is mutual information between latent facts and model outputs, together with Fano-style lower bounds on semantic error. We clarify the stochastic assumptions required for non-degenerate information measures, distinguish true data-generating uncertainty from model-implied uncertainty, and replace unsupported hard capacity formulas with explicit capacity surrogates tied to precision, context budget, and effective representational rank. Under standard identification assumptions, we derive a baseline bound \begin{equation*} H_R \geq \max\left\{0,\,1-\frac{I(F;Y)+1}{\log M}\right\}, \end{equation*} where is hallucination rate, is the latent semantic fact, is model output, and is semantic cardinality. We also provide a distribution-dependent variant and a bottleneck-aware extension for retrieval-augmented generation. This paper contributes a mathematically consistent formulation, a tighter assumptions section, and concrete empirical protocols for estimation and falsification.