مراجعة PREreview لMeasurement Reliability Confounds Cross-Regime Comparisons of Postprandial Glucose Prediction: Evidence from Bidirectional Cross-Cohort Transfer
- نُشرت
- DOI
- 10.5281/zenodo.24389074
- الترخيص
- CC BY 4.0
The study, Measurement Reliability Confounds Cross-Regime Comparisons of Postprandial Glucose Prediction, investigates how measurement reliability influences the performance and generalizability of machine-learning models predicting postprandial glucose responses (PPGR).
Using two independent continuous glucose monitoring (CGM) datasets (CGMacros and BIG IDEAs), the authors reproduce an existing prediction model, evaluate its performance within each cohort, and conduct bidirectional cross-cohort validation.
The key findings are:
1. Measurement reliability strongly influences prediction performance. Duplicate-meal analysis revealed substantial differences in repeatability depending on meal type, with reference breakfasts showing higher reliability (r = 0.799) than free-choice dinners (r = 0.374) and snacks (r = 0.376).
2. Poor cross-cohort generalizability. The model showed substantially reduced performance when transferred between cohorts. Prediction of incremental glucose response (iAUC) declined to r = 0.227 in forward transfer and r = 0.085 in reverse transfer, suggesting limited portability across different measurement and data-collection regimes.
3. AUC can overestimate predictive performance. The authors demonstrate that total area under the glucose curve (AUC) is strongly influenced by pre-meal glucose levels. Consequently, AUC may reflect baseline glycemic status rather than the actual meal-induced glucose excursion. iAUC provides a more appropriate measure for evaluating meal-specific glucose responses.
4. Aggregation does not necessarily indicate better meal-level prediction. Although averaging predictions at the individual level improved within-cohort correlations, it did not rescue cross-cohort iAUC prediction.
Contribution to the field: This work highlights an important but frequently overlooked limitation in personalized nutrition research: differences in measurement reliability can confound comparisons of predictive model performance across studies and populations. It emphasizes the need to report measurement reliability alongside prediction metrics, distinguish meal-level from person-level prediction, and evaluate models under comparable measurement conditions. These findings provide useful methodological guidance for developing and validating more reliable and generalizable PPGR prediction.
Major issues
1. Measurement reliability is not established in both cohorts. The study estimates repeatability using CGMacros but does not measure it directly in BIG IDEAs. Consequently, the central argument that reliability explains cross-cohort performance differences remains partly inferential rather than experimentally demonstrated.
2. Multiple confounding factors limit interpretation. Device type, meal annotation methods, and participant characteristics differ simultaneously between the two cohorts. The study cannot independently determine how much each factor contributes to the observed decline in prediction performance.
3. Limited sample size and external validation. BIG IDEAs includes only 16 participants. Although bidirectional validation strengthens the analysis, the relatively small number of individuals limits statistical power and the generalizability of the conclusions.
4. Limited evidence for the proposed reliability framework. The comparison with previous studies is informative, but the reliability estimates come from different populations and experimental protocols. In particular, the use of an external device-level reliability estimate as a proxy for BIG IDEAs introduces uncertainty into the comparison.
5. Clinical applicability remains uncertain. The study demonstrates limitations in meal-level prediction and reports substantial variability in prediction intervals across individuals. However, it does not establish whether the proposed framework or prediction results can improve personalized dietary recommendations or clinical decision-making.
Minor issues
1. Clarify terminology and abbreviations. The distinction between AUC, iAUC, peak glucose rise, and measurement reliability could be explained more consistently, particularly for readers less familiar with CGM-based glucose prediction.
2. Improve presentation of statistical uncertainty. Confidence intervals should be reported consistently alongside key performance metrics, especially for cross-cohort correlations and reliability estimates.
3. Improve visualization of cohort differences. A concise comparison table highlighting differences in participant characteristics, CGM devices, sampling frequencies, and meal annotation protocols would improve readability.
4. Expand discussion of alternative explanations. The discussion could more clearly distinguish the effects of measurement reliability from differences in population characteristics, feature distributions, model architecture, and meal-recording accuracy.
5. Improve the discussion of practical implications. The recommendations for reporting reliability and evaluating individual-level prediction intervals are valuable, but more specific guidance on implementing these recommendations in future PPGR studies would strengthen the manuscript.
Competing interests
The author declares that they have no competing interests.
Use of Artificial Intelligence (AI)
The author declares that they did not use generative AI to come up with new ideas for their review.