PREreview of Improving Medicare Fraud Detection Accuracy in Deep Learning by Exploring Feature Selection and Data Sampling Techniques

This paper investigates whether combining feature selection (Chi-Square, Mutual Information) with data sampling techniques (RUS, ROS, SMOTE) improves deep learning accuracy for Medicare fraud detection on the widely-used Kaggle Healthcare Provider Fraud Detection dataset. The authors train a simple sequential neural network under seven configurations and report that combining Chi-Square feature selection (top 25 features) with SMOTE oversampling achieves 95.4% accuracy with balanced 0.94 F1-score across both classes. The proposed combined pipeline is compared against a baseline deep learning model and individual feature-selection and sampling techniques applied in isolation.

The paper has several genuine strengths that should be preserved through revision:

• Reproducibility. The authors provide both a GitHub repository and a versioned Zenodo DOI for the code, which is above the norm for this literature and enables independent verification.

• Clear experimental structure. Table 1's seven-row comparison isolates the contribution of each technique cleanly, allowing readers to see incremental effects of feature selection alone, sampling alone, and the combination.

• Practical relevance. The focus on minimizing false negatives (Recall of 0.98 for the Fraud class) is well-aligned with real-world fraud-detection priorities where missed fraud carries significant financial and audit consequences.

• Public benchmark dataset. Using the Kaggle CMS-derived dataset allows direct comparison with prior work in the literature.

• Learning curve analysis. Figure 6 provides evidence of low overfitting, which is often omitted in similar papers.

Major issues

1. Accuracy is the wrong headline metric for imbalanced fraud detection. The paper leads with accuracy throughout the abstract, results section, and conclusion. For a dataset with a 61.6/38.4 class split and particularly for fraud-detection use cases where operational value depends on precision at low false-positive rates accuracy is a weak and potentially misleading metric. The paper should foreground AUROC, AUPRC (average precision), precision-recall curves, and F1-score as primary metrics. AUPRC is especially important in this domain because it directly reflects the trade-off operators face when tuning alert thresholds. Table 2 provides some of these metrics for the proposed model only; they should be provided for all seven configurations to make the comparisons meaningful.

2. No statistical significance testing or confidence intervals. The reported accuracy differences (0.920 → 0.954) are presented as if they were reliable measurements, but no confidence intervals, cross-validation folds, or repeated runs with different random seeds are reported. A single train/test split with a stochastic training procedure (SMOTE synthesis, neural network initialization, mini-batch order) can easily produce ±1–2% variation on this dataset. Without k-fold cross-validation or bootstrap intervals, the reader cannot distinguish real improvement from run-to-run noise. Please report mean ± SD across at least 5 runs (or ideally 5-fold or 10-fold CV) for each configuration.

3. SMOTE applied before train/test split is a common data-leakage pitfall — the paper does not clarify. The methodology does not explicitly state whether SMOTE oversampling was performed before or after the train/test split. If SMOTE was applied to the full dataset before splitting, synthetic samples derived from test-set neighbors leak into training, artificially inflating test-set performance. This is a well-documented failure mode (see Santos et al., 2018; Kovács, 2019) and can account for multi-percentage-point overestimates. Please explicitly confirm that SMOTE was applied only to the training fold, and clarify this in the methodology.

4. The "combined model" does not actually outperform SMOTE alone. Table 1 shows SMOTE-only accuracy = 0.957 and the proposed Chi-Square + SMOTE combined model = 0.954. The combined model is 0.3% worse. The paper frames this as "remarkably similar" and selects the combined model on the basis of "reduced overfitting and consistently stable accuracy," but no quantitative evidence of the overfitting difference between the two configurations is provided (Figure 6 shows only the proposed model). To justify preferring the combined model, please provide learning curves for SMOTE-only vs. the combined model, or report validation-loss gaps, so the "reduced overfitting" claim is empirically supported rather than asserted. As written, a reasonable reader could conclude that feature selection provided no benefit or even slight harm on this task.

5. Weak baselines. The "baseline" is the same deep learning model without feature selection or sampling. This does not test whether deep learning is even the right approach for this problem. Prior literature on this exact Kaggle dataset routinely reports XGBoost, Random Forest, and LightGBM performance in the 92–96% AUC range with much lower computational cost. Please add at least one strong classical ML baseline (XGBoost recommended) evaluated under identical feature selection and sampling conditions, so the deep-learning contribution can be assessed on its merits.

6. Deep learning architecture is under-specified. Section 3.4 states a Keras Sequential model with "a few hidden layers," ReLU activation, and sigmoid output, but does not specify: number of hidden layers, layer widths, optimizer, learning rate, batch size, number of epochs, loss function, regularization (dropout, L2), early stopping criteria, or validation-split ratio. Without these details, the results are not reproducible even with the code available (code alone doesn't communicate design rationale). Please add a hyperparameters table.

7. Chi-Square selecting the same features as intuition — worth interrogating. Figure 3 shows the top Chi-Square features are dominated by claim reimbursement amounts (means and standard deviations). Since these are almost tautologically correlated with fraud in this dataset (fraudulent providers bill more), this raises a question about whether the "learned" model is capturing genuine fraud patterns or exploiting a leaky proxy. Please discuss feature-level interpretability and consider ablating the top 2–3 reimbursement features to test whether the model still performs well on subtler signals.

8. External validity is unaddressed. All results are from a single dataset (Kaggle/CMS), which is itself a subset with known selection biases. The Limitations section briefly acknowledges this, but the paper's conclusions ("outstanding ability to detect fraudulent claims," "game-changer") over-generalize. The claims should be scoped to "on this benchmark dataset" throughout.

9. Mutual Information anomaly needs explanation. Section 3.2.2 reports that "all the features got the same importance score of 0.024503." Identical MI scores across all features is highly unusual and suggests either (a) a computational bug, (b) a normalization artifact, or (c) SelectKBest returning uniform values when passed inappropriate input types (e.g., continuous features treated as categorical). Please investigate and correct or explain this result. As presented, the MI results are uninterpretable.

10. Blockchain integration in Future Work is speculative and unfocused. The conclusion introduces blockchain as a "game-changer" for future work, but the paper offers no analysis of how blockchain would integrate with the proposed pipeline, what threat model it would address, or why it is more suitable than other data-integrity approaches (e.g., cryptographic audit logging, immutable append-only databases). This paragraph reads as tacked-on and could be shortened or refocused on more directly relevant future work (concept drift handling, adversarial robustness, deployment latency).

Minor issues

• Class distribution inconsistency. Section 3.1 states approximately 350,000 claims are "Not Fraud" and >200,000 are "Fraud" a roughly 63/37 split. Section 3.3 states 61.6% / 38.4%. Section 3.1 also says the dataset has 558,212 claims. These need to be reconciled — the dataset is provider-level after aggregation, not claim-level, and the figures suggest the label is being applied at different granularities in different sections.

• Figure 1 caption grammar. "The claims distribution are categorized" → "The claims are categorized"

• Table 1 header should include additional metrics (AUROC, AUPRC, F1) not just accuracy, as noted above.

• Figure 3 y-axis label is missing units and is this Chi-Square statistic value, p-value, or feature importance rank?

• Reference [1] cites a Department of Justice press release for a "1.4 billion" loss figure. HHS OIG and CMS also publish more comprehensive figures I would recommend citing the annual Health Care Fraud and Abuse Control Program report.

• Reference [16] is an arXiv preprint on ad event prediction being used as the citation for both Chi-Square and Mutual Information formulas. Standard textbook citations (e.g., Guyon & Elisseeff 2003 for feature selection foundations) would be more authoritative for these standard techniques.

• The Kaggle dataset is often referred to as the "CMS DE-SynPUF" or the 2018 Kaggle Healthcare Provider Fraud dataset please add its provenance chain (original CMS source → Gupta's Kaggle upload) rather than citing only the Kaggle mirror.

• Several sentences use informal language: "a hot research area," "game-changer," "sparks a set of recommendations." Consider more formal academic phrasing.

• Table 1 layout: consider bolding the highest value in each metric column for reader ease.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.