PREreview de What Does an LLM Learn from Reinforcement Learning? A Mechanistic Interpretability Perspective with Fixed-SAE Track

Summary

The authors tackle a real obstacle in interpretability: you can't compare SAE features across training checkpoints, because separately trained dictionaries assign feature indices arbitrarily. Their fix is simple and sensible — pool activations from the base model and every RL checkpoint, train one SAE on the lot, freeze it, and re-encode each checkpoint through that fixed dictionary. Applied to GRPO on Qwen2.5-1.5B-Instruct, they find RL changes surprisingly little: drift is small, gradual, concentrated in late layers, and centered on about ten features that fire on formatting scaffolds — step breaks, LaTeX delimiters, answer wrappers — rather than problem content. Steer those directions into the base model and you get most of RL's measured gain back.

Introduction

Clear. The setup works well: behavioral accounts of RL contradict each other, all of them grade outputs rather than looking inside, SAEs are the obvious tool, and existing SAE methods can't handle checkpoints. The objective follows naturally — what does RL give a model at the representation level, and can it create genuinely novel features?

Methods — somewhat appropriate

The core idea is sound, and the appendix is unusually thorough: four RL algorithms, five SAE architectures, three model scales, placebo feature sets, random-direction controls, and a two-grader protocol whose decomposition is actually validated in Table 3 rather than assumed.

Four things hold it back:

1. The synthetic benchmark is missing. The abstract and introduction both promise a benchmark with features known by construction, to test whether RL instills novel features. It isn't in Section 4 or anywhere in the appendix. This isn't a detail — novel-feature detection is half the paper's stated question and the main reason the pooled design is supposed to beat SAE Track.

2. FVU sits around 0.52. Half the activation variance is unexplained. The authors rightly note it's stable, which rules out dictionary degeneration, but never ask whether reconstruction this weak supports claims about specific feature identity.

3. Amplified features replicate worse than the framing suggests. Cross-algorithm agreement is reported as Spearman 0.42, but top-K Jaccard is 0.28 for amplified features versus 0.61 for suppressed. The amplified set carries the headline finding and the whole steering intervention.

4. Causal claims rest on one 1.5B model. The authors say plainly that ablations are inconclusive at 3B and 7B. Honest, but limiting.

Conclusions — somewhat supported

The main finding holds up from several directions at once. The formatting signature appears whether features are ranked by endpoint drift or by reward correlation. The elicitation claim rests on three independent results: steering reproduces the effect, all 29 RL-fixed problems are base-solvable within 64 samples, and few-shot prompting substitutes for RL on the base model while adding nothing to the RL model.

The authors also qualify carefully. They don't claim RL never improves reasoning — sustained training does help. And their handling of the reasoning set R is the best reasoning in the paper: it's causally necessary but can't be injected as a static vector, and rather than smoothing that over they conclude reasoning depends on dynamically sustained computation.

Two problems:

1. "Around 80%" doesn't come from anywhere. At α = 10, steering gives strict 0.192 and tag-rate 0.600 against RL-final's 0.175 and 0.533. Steering exceeds RL on both. No calculation in the paper yields 80%, and the conclusion says "most" instead.

2. Steering rests on n = 120 with overlapping intervals. The α = 10 strict interval is [.131, .271], RL-final's is [.117, .253]. "Matches within confidence intervals" is true but thinner than it sounds, especially since the steering features were selected on the same trajectory they're tested against.

Figures — somewhat appropriate and clear

Figure 3 is the paper's best: the inverted-U with RL reference lines shows the match and the off-manifold collapse in one glance. Table 7's Wilson intervals are the right call, and Table 3 lets a reader audit the grader decomposition rather than trust it.

But several figures are unreadable at page size. Figure 7 is the worst — three panels with legends listing dozens of run names in microscopic type, conveying essentially nothing. Figure 2's 24 stacked labels, Figure 6's overlapping trajectories, and Figure 9's axis labels all have the same problem.

Discussion — somewhat clear

Strong where it explains mechanism. The timing result (86% of amplification done within 50 steps) turns the elicitation story into something with a shape. Re-characterizing the apparent collapse as reward-hacking brevity — checked against held-out generation length and R's stability — is the right diagnostic instinct. And keeping the superseded steering analysis in A.12, with an explanation of why it misled, is a level of transparency most papers skip.

Weak where it should consolidate. The conclusion recaps without revisiting the reproducibility asymmetry, the inconclusive larger-scale results, or the missing benchmark. There's no limitations section, and next steps amount to "future work at larger scales" — not even the benchmark they already promised.

Contribution — somewhat likely

The method is genuinely reusable and generalizes past RL to any training-trajectory analysis. The findings land in a live dispute, and the convergent evidence for elicitation is more convincing than any single result would be.

What limits it: one small model, one reward function that explicitly requires an answer wrapper — so "RL amplifies formatting" may partly be a finding about this reward rather than RL — and the unanswered half of the framing question.

Language

Clear and precise. Two things to fix: the Acknowledgments section is empty, and A.13 has an unresolved citation ("the unbiased estimator of ?"). Minor slips elsewhere ("RL genuinely incentivize," "limitedly understood") don't impede reading.

Recommendation

Worth reading if you work on interpretability of training dynamics or on what RLVR actually does. The method is useful and the elicitation evidence is well triangulated.

Three things before it's ready:

1. Include the synthetic benchmark or drop the claim. Right now the abstract advertises an experiment the paper doesn't contain.

2. Fix or remove "around 80%." Steering beats RL-final on two of three metrics; whatever 80% refers to needs deriving.

3. Add a limitations section and a conclusion that folds in the reproducibility asymmetry and the scale-limited causal evidence.

Larger n for steering, held-out feature selection, and an RL run whose reward doesn't require a format wrapper would all strengthen it further.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.