PREreview de Protein contacts are already in the attention: a single-forward-pass alternative to the Categorical Jacobian

Protein contacts are already in the attention: a single-forward-pass alternative to the Categorical Jacobian

In this preprint Thorstenson argues in favor of selecting attention heads to predict contacts from protein language models. He specifically contrasts his “fusion” approach, where attention heads are ranked by precision on a set of training proteins and the top k are selected, with the so-called “categorical Jacobian” (CJ) operation from [1], which was developed as a label-free method to extract pairwise coupling weights from pLMs.

The main claim in the headline is a basic result that has been known since the very first pLMs and was already standard when the categorical Jacobian was proposed. The CJ was not proposed specifically as a general contact prediction tool and the implied comparison is moot. In [1] it was used as an interpretability tool to investigate pLM innards without relying on any supervision, structural or otherwise.

Accounting for this conceptual issue, the work presents a variation on one of the ablation tests in [2], where the authors asked whether simply averaging the top-{1,5,10} attention heads was enough for contact prediction. The choice of heads in that case came from ranking logistic regression weights and then averaging the top-weighted heads, with the regression coming from 20 training proteins. The claimed advance is that picking attention heads by precision ranking on 10 proteins suffices, which is shown. Thorstenson states that his fusion method improves on the Rao averaging method by having a smaller selection budget, but he actually compares his fusion to a fitted logistic regression rather than the top-k averaging. The claim advantage over Rao’s top-k approach is therefore not demonstrated.

Thorstenson also assembles a new set of 29 evaluation structures deposited post-ESM-2 training cutoff and demonstrates reasonable precision improvements on this set for his fusion method vs. the CJ or the single top attention head. This is a useful result but the small number of sequences is limiting. There is not a matched comparison to the top-k attention averaging approach in [2] either.

A useful formulation is also presented, the Representation-CJ. This is a generalization of the CJ where hidden state perturbations rather than output logits are measured which allows an analogous construction to be applied to broader models. The agreement of the two is convincingly demonstrated.

As a final note, there are some citation issues which may be related to the stated use of AI tools.

1. Cited as:

Lennard Hermann, Tobias Fiedler, Hieu A. Nguyen, Magdalena Nowicka, and Jakub M. Bartoszewicz. Beware of data leakage from protein LLM pretraining. In Proceedings of the 19th Machine Learning in Computational Biology meeting (MLCB), volume 261 of Proceedings of Machine Learning Research, pages 106–116, 2024.

Wrong name. correct cite:

Leon Hermann, Tobias Fiedler, Hoang An Nguyen, Melania Nowicka, Jakub M Bartoszewicz. Beware of Data Leakage from Protein LLM Pretraining. Proceedings of the 19th Machine Learning in Computational Biology meeting, PMLR 261:106-116, 2024.

1. Cited as:

Jakub M. Bartoszewicz, Hieu A. Nguyen, Lennard Hermann, Tobias Fiedler, and Magdalena Nowicka. A flaw in using pretrained protein language models in protein–protein interaction inference models. Nature Machine Intelligence, 2025. doi: 10.1038/s42256-025-01176-7.

True cite:

A flaw in using pretrained protein language models in protein–protein interaction inference models. Joseph Szymborski & Amin Emad

1. Cited as:

Anonymous. LiveProteinBench: A contamination-free benchmark for assessing models’ specialized capabilities in protein science. arXiv preprint arXiv:2512.22257, 2025.

Real cite:

LiveProteinBench: A Contamination-Free Benchmark for Assessing Models' Specialized Capabilities in Protein Science. Dingyi Rong, Zijian Chen, Qi Jia, Kaiwei Zhang, Haotian Lu, Guangtao Zhai, Ning Liu

References

[1] Z. Zhang, H.K. Wayment-Steele, G. Brixi, H. Wang, D. Kern, & S. Ovchinnikov, Protein language models learn evolutionary statistics of interacting sequence motifs, Proc. Natl. Acad. Sci. U.S.A. 121 (45) e2406285121, https://doi.org/10.1073/pnas.2406285121 (2024).

[2] Transformer protein language models are unsupervised structure learners. Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, Alexander Rives. bioRxiv 2020.12.15.422761; doi: https://doi.org/10.1101/2020.12.15.422761

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.