Aller directement au contenu principal

Rédiger un PREreview

Origin-Paged Memory: Treating Language-Model Context as Virtual Memory A Pre-Registered First Test of the Origin-Paged Attention Program on a 4B-Parameter Model

Publié
Serveur de preprints
Preprints.org
DOI
10.20944/preprints202611.1367.v1

Language-model agents that work for hours, or read documents far longer than their context window, need a principled way to decide what stays in context. We study Origin-Paged Attention (OPA), a research program that treats context like virtual memory in an operating system. The archive is split into fixed-size chunks called origins. The program has two layers. Its memory layer, Origin-Paged Memory (OPM), keeps a few origins in working slots, pins exact evidence, and pages an earlier origin back in when an entity becomes relevant late, using an index of the entities each origin mentions. Its attention layer, Origin-Anchored Attention (OAA), would move the same idea inside the model by giving every origin its own positional frame and an address in hyperbolic space. We prove that OAA attention factors exactly into a choice of origin times ordinary attention within it, so the error caused by paging equals the attention mass that was paged out. We then test 13 pre-registered hypotheses on the open Qwen3-4B-Instruct model, with every verdict rule fixed before any test data existed. On synthetic late-binding multi-hop questions at 32K tokens, OPM answered 76.6% correctly, against 62.4% for one-shot retrieval (a gain of 14.2 points, 95% CI 9.2–19.1); it also beat a gist-memory agent by 39.6 points, and pinning cut repeated page-ins by 2.54 per item. A graph of links between origins, hyperbolic distances, and a “red-dot” prefix that carries the end of one origin into the next added nothing measurable. At the attention level, a short fine-tuning run did not teach OAA to attribute facts to their origin (31.2% against 99.6% for standard attention), so the remaining attention-level and systems tests were not run. The memory layer is supported; moving it inside attention remains open.

Vous pouvez rédiger un PREreview de Origin-Paged Memory: Treating Language-Model Context as Virtual Memory A Pre-Registered First Test of the Origin-Paged Attention Program on a 4B-Parameter Model. Un PREreview est une évaluation d'un preprint et peut varier de quelques phrases à un rapport détaillé, semblable à un rapport d'évaluation par les pairs organisé par une revue.

Avant de commencer

Nous vous demanderons de vous connecter avec votre identifiant ORCID iD. Si vous n'en avez pas, vous pouvez en créer un.

Qu’est-ce qu’un ORCID iD ?

Un ORCID iD est un identifiant unique qui vous distingue de toute personne ayant le même nom ou nom similaire.

Commencer maintenant