PREreview de A Framework for Context Enrichment and Selective Robustness in LLM Social Agents

My training is in statistics and decision making. I reviewed this preprint with that background, looking at how the comparisons are built, how the outcomes are summarized, and whether the conclusions follow from the estimates.

This study focuses on large language models used as agents in social simulations, and on the fact that those agents often change their actions when a prompt is reworded even though the social situation is meant to stay the same. The authors separate context enrichment into five substitutions: a minimal prompt, neutral padding, generic social background, descriptive decision grounding, and normative decision grounding. They evaluate these with selective robustness, meaning stability under surface changes that are intended to preserve meaning, together with a prespecified change in behavior when the social world changes. The demonstration is one Prisoner’s Dilemma-like hospital supply decision. The agent chooses TRANSFER or RETAIN, the payoff table is fixed at T = 5, R = 3, P = 1, S = 0, and the two worlds differ in whether supply continuity is covered by a shared regional arrangement or by separate local arrangements. These worlds are crossed with the five context conditions and four surface forms, and gpt-5.6-sol and gpt-5.6-terra are each called ten times per cell (800 calls). They found that 1) descriptive and normative grounding produce large regional-minus-local gaps in transfer probability for both models, about 86 and 92 percentage points for Sol and about 57 and 77 for Terra (Table 2), 2) on the authors’ mean surface-sensitivity measure, Sol’s grounded conditions are about 4 percentage points while Terra’s remain about 13 to 15, 3) neutral padding reduces the world gap relative to the minimal prompt, and generic social background sends the two models in different directions, and 4) the pattern is not the same for the two models. The patterns in Tables 2 and 3 are supported by the reported estimates. The further claim that this design identifies which forms of enrichment produce more desirable and robust behavior is not yet supported, for the reasons below. The prompts and the code remain useful for later checks of this kind.

Major comments

Abstract, and Section 4: “showing how it can be used to identify which forms of context enrichment produce more desirable and robust agent behavior.” Section 4 later says that “the grounded prompt conditions add more detail about the social-world distinction than the non-grounded conditions, so the experiment does not fully separate the effect of decision grounding from increased salience of that distinction.” The second statement is the one the design can carry. Every condition already contains the regional or local world fact. Descriptive grounding adds that the shortage procedure “covers your decision,” and normative grounding adds that “your role requires you to follow” that procedure (Appendix B.2). The larger gaps in Table 2 are real. They are also what would be expected if the model were responding to the world fact being repeated and tied more closely to the choice. I recommend adding a control that restates or highlights the world contrast to a similar degree, without the relationship, the procedure, or the decision link. I also recommend rewriting the abstract so that it stays with what Section 4 already allows.

Figure 1, Table 3, and Appendix B.3. Surface sensitivity “is the mean absolute difference between the canonical prose cell mean and each of the other three surface cell means,” and “the surface sensitivity measure averages the regional and local values.” That average is why Sol’s normative condition is reported as 3.9 percentage points and sits in the upper left of Figure 1. In Table 3, the regional normative cells are 96.6% transfer for canonical prose and 77.6% for bullets, which is a gap of about 19 points. All four local cells are under 0.1%, so the local world pulls the average down. Terra’s normative value of 15.0 percentage points is concentrated in one place: local alternative prose is 89.9%, and the other three local surfaces are at most 0.9%, while the regional surfaces are all essentially 100%. The local descriptive row has a similar split, 84.1% against 19.0% to 36.5%. I recommend reporting the largest absolute surface gap inside each world next to the current mean, marking those cells in Table 3, and treating a condition as surface-stable only when the gap stays small in a world that is not already stuck at 0% or 100%.

Appendix B.2 and B.3. I am not convinced the plotted quantities are yet the surface invariance and world responsiveness defined in Section 2.2. My concerns are below.

The paper says the four surface forms are meant to preserve meaning, and Section 4 states that no independent reader checked this. Alternative prose changes clause order and verbs, and it moves the requirement sentence to the front. That is the cell in which Terra goes from at most 0.9% to 89.9% transfer. “Covers your decision” and “your role requires you to follow” are also close enough that I would want a reader to judge whether normative force is confined to the normative condition. Appendix B.2 says gpt-5.6-sol assisted in drafting the prompt content and the surface variations, and Sol is the model that then shows the smaller surface score. I recommend an independent reading of every surface pair, and a rerun of the grounded cells with paraphrases drafted without the evaluated model.

“We noticed after data collection that the provider had used a default top-p of 0.98, causing one action candidate to be omitted in 339 of the 800 calls.” The midpoint of the omitted mass was used. The authors report that varying those values inside their bounds moves aggregate measures by no more than 0.49 percentage points and preserves the rankings. The intervals in Figure 1 still come from resampling the ten calls within a cell. They do not reflect uncertainty about wording, and cells near the floor or the ceiling produce very narrow intervals, such as Sol’s normative responsiveness of 92.0 [91.0, 92.9]. I recommend extracting both action tokens with top-p set to 1, and stating next to Figure 1 that the bars are call-level bootstrap intervals.

Section 3 and the fixed decision block in Appendix B.2. The authors “expected more transfer in the regional world because its emergency-supply arrangement covers the supply needs of both hospitals.” The payoff ordering is a standard Prisoner’s Dilemma, so RETAIN dominates, but the procedure text never says that TRANSFER is required. The same fixed block says: “The hospitals have an ongoing operational relationship, and similar allocation situations may arise in the future.” That sentence is shared by every cell, so it does not create the regional-minus-local gap, but it does change what decision the agent is being asked to make. I recommend stating the dominant-strategy benchmark in the paper, specifying the action implication of the shortage procedure in the prompt, and either removing that future-allocation sentence or crossing it with the world factor. Human choices under the same prompts, which the authors already propose, would show whether a gap of this size is a plausible response.

Minor comments

Section 2.1: the ordering “does not serve as a gradient of longer nor necessarily ‘richer’ information.” The sentence is hard to parse. Each enriched condition adds about three sentences, so the design is roughly matched on length. I recommend replacing it with a direct statement that the conditions are matched on the number of added facts and are not ordered by length.

The condition labels are not the same in Figure 1, Table 2, and the prose (“Generic social background” and “Generic background,” and the same shortening for both grounding conditions). I recommend using one name for each condition throughout.

Figure 1 does not say how the intervals were computed, and it uses “upper left” without a cutoff. Appendix B.3 says the intervals are 95% percentile intervals from resampling the ten calls 10,000 times. I recommend putting that in the caption, noting that a floor or ceiling in one world shrinks the averaged surface score, and either defining the preferred region or discussing the two measures without “upper left” and “more desirable.”

The prespecified direction has a long history in human experiments that change the label of a Prisoner’s Dilemma and leave the payoffs fixed. That literature is not cited, and neither is most of the work on language models in dilemma games beyond Lorè and Heydari (2024). I recommend adding those citations. One accumulated-persona condition would also connect the substitution design to the longer-biography practice discussed in the introduction.

Appendix B.2 prints the prompts with line-wrap arrows. I recommend a monospace listing, and, with the API settings, the model snapshot date, whether a system message was used, and the exact conversion from log-probabilities to a transfer probability.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.