Skip to main content

Write a PREreview

Origin-Paged Memory: Treating Language-Model Context as Virtual Memory A Pre-Registered First Test of the Origin-Paged Attention Program on a 4B-Parameter Model

Posted
Server
Preprints.org
DOI
10.20944/preprints202611.1367.v1

Language-model agents that work for hours, or read documents far longer than their context window, need a principled way to decide what stays in context. We study Origin-Paged Attention (OPA), a research program that treats context like virtual memory in an operating system. The archive is split into fixed-size chunks called origins. The program has two layers. Its memory layer, Origin-Paged Memory (OPM), keeps a few origins in working slots, pins exact evidence, and pages an earlier origin back in when an entity becomes relevant late, using an index of the entities each origin mentions. Its attention layer, Origin-Anchored Attention (OAA), would move the same idea inside the model by giving every origin its own positional frame and an address in hyperbolic space. We prove that OAA attention factors exactly into a choice of origin times ordinary attention within it, so the error caused by paging equals the attention mass that was paged out. We then test 13 pre-registered hypotheses on the open Qwen3-4B-Instruct model, with every verdict rule fixed before any test data existed. On synthetic late-binding multi-hop questions at 32K tokens, OPM answered 76.6% correctly, against 62.4% for one-shot retrieval (a gain of 14.2 points, 95% CI 9.2–19.1); it also beat a gist-memory agent by 39.6 points, and pinning cut repeated page-ins by 2.54 per item. A graph of links between origins, hyperbolic distances, and a “red-dot” prefix that carries the end of one origin into the next added nothing measurable. At the attention level, a short fine-tuning run did not teach OAA to attribute facts to their origin (31.2% against 99.6% for standard attention), so the remaining attention-level and systems tests were not run. The memory layer is supported; moving it inside attention remains open.

You can write a PREreview of Origin-Paged Memory: Treating Language-Model Context as Virtual Memory A Pre-Registered First Test of the Origin-Paged Attention Program on a 4B-Parameter Model. A PREreview is a review of a preprint and can vary from a few sentences to a lengthy report, similar to a journal-organized peer-review report.

Before you start

We will ask you to log in with your ORCID iD. If you don’t have an iD, you can create one.

What is an ORCID iD?

An ORCID iD is a unique identifier that distinguishes you from everyone with the same or similar name.

Start now