Skip to PREreview

PREreview of The Prosocial Turn in Video Games: A Twenty-Year Shift in the Motivations Games Simulate

Published
DOI
10.5281/zenodo.24348038
License
CC BY 4.0

In “The prosocial turn in video games: a twenty-year shift in the motivations games simulate”, Javelle et al. (2026) examine whether the motivational composition of video games has shifted (from competitive towards prosocial) between 1993 and 2016. I read the manuscript with interest and commend the authors for engaging with the topic. The idea of examining games’ content as “cognitive fossils” is an exciting one. However, I have some issues with the work that I’d like to see fixed before I can recommend the manuscript.

1 LLM annotation

Authors used large language models (LLM) to code each of 11k+ games on 48 motivational dimensions. Something like this should be an exemplary use of these AI tools, and I appreciate the authors’ engaging with this. However, it would be prudent to examine this procedure’s measurement validity in greater detail both due to its novelty and the controversy surrounding the broader topic of LLMs in research. I appreciate the authors’ validation efforts, but think that due to novelty this should be expanded on.

First, authors note that “This LLM-based approach has also been used and validated in previous applications of the motivational-ingredient taxonomy (Dubourg et al., 2024; Dubourg, Thouzeau, et al., 2026; Dubourg & Thouzeau, 2026).” The first reference is to a tutorial, the second to a discussion of the motivational ingredients framework, and so neither is directly relevant here. The third is cited as “Manuscript submitted for publication” without a link/DOI, so I cannot find prior work evaluating the method’s validity.

To code a single game, authors instruct an LLM to evaluate {game_title} ({year}, {platform}) on its 48 motivational dimensions. My questions here include but are not limited to:

  1. What information does the LLM use when presented with such a query?

  2. Does {game_title} activate some “knowledge” about this game, and this game only?

  3. How sensitive are the annotations to variations in the prompt (excluding the motivational ingredients), such as including/omitting the year, platform, and assigned expertise?

  4. There is presumably more information about newer games in the model’s training data, whereas old games’ annotations might rely more on e.g. {year}, {platform}, or some indirectly related info like game genre; how would these impact the results?

  5. Reliability is checked against a single author’s codings, but optimally would be checked against more than one human rater (who should also be checked between each other for reliability). How would these compare to e.g. (expert) players’ ratings?

  6. What do the full (motivational) item-level agreement results look like? A table here could be helpful.

2 Data: Terms of use & ethics

I have some concerns over how the data were obtained and how they are cited (and shared). These should be clarified before publishing the manuscript.

  1. Please describe how use of the VGChartz data adheres to VGChartz’s terms of service / licensing? I am not a lawyer but it looks like e.g. scraping the site for data would go against VGChartz’s TOS (https://www.vgchartz.com/terms-of-use.php). What about other related ethical/legal requirements?

  2. I would also want to know how the VGCharz data was obtained? I cannot find an explanation of the procedures for (e.g.) scraping the data, and so am unable to evalute whether this method is valid. For example, if it was scraped, (reproducible) scripts for this could be shared.

  3. Moreover, though I strongly advocate open data, it is unclear whether authors would be allowed to share the VGChartz data (and under what license). This should be clarified.

3 Data: Scope

Authors analyzed 11,359 games between 1993 and 2016. Presumably this was due to a (unjustified) minimum of 30 games per year requirement, but even that seems unnecessary. On a quick glance, there are 67k games on VGChartz from 1988 to the present year. Without a reasonable justification for this data restriction (when in principle it doesn’t seem impossible to scale up to 67k titles) I cannot recommend the work for publication in its present form.

  1. How do the authors justify the restrictions, and how confident are we that the findings would hold for the past decade?

4 Reproduction check

I was unable to reproduce the analyses: The reproduction instructions in README.md are wrong (e.g. the files referred to in “## How to reproduce” do not exist). Data files have the ‘comma separated values’ (.csv) extension but are instead tables of semicolon separated values. README.md is not plain-text formatted (as is recommended) and thus difficult to read. Authors should share the code and data that was used to produce the results, and ensure these are clearly and concisely documented, and can be shown to lead to presented results.

Thanks for this exciting work and I hope to have provided you with useful comments for potentially improving your manuscript.

Signed, Matti Vuorre

References

Javelle, Alexandre, Benjamin Dahan, Toni Z A Ali, Valérian Chambon, and Edgar Dubourg. 2026. The Prosocial Turn in Video Games: A Twenty-Year Shift in the Motivations Games Simulate. 5ebuq_v1. PsyArXiv. https://osf.io/preprints/psyarxiv/5ebuq_v1/.

Declarations

This review of Javelle et al. (2026) is contributed by Matti Vuorre under CC-BY to the PREreview platform. I didn’t use LLMs in drafting this comment.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.