Reproducibility Meta-Analysis of Divergent Llama-3.1-8B Ruler Benchmarks Across Four Independent Studies

Abstract

Abstract: This paper presents a comprehensive systematic review of generative models (GANs, VAEs, DMs, and LLMs) used to synthesize various medical data types, including imaging (dermoscopic, mammographic, ultrasound, CT, MRI, and X-ray), text, time-series, and tabular data (EHR). Unlike previous narrowly focused reviews, our study encompasses a broad array of medical data modalities and explores various generative models. Our aim is to offer insights into their current and future applications in medical research, particularly in the context of synthesis applications, generation techniques, and evaluati

Research Question

Reproducibility meta-analysis: 4 independent publications report divergent Llama-3.1-8B performance on Ruler with a 83.7 percentage-point spread (range 1.9%–85.6%). Source papers: "Ruler Score Discrepancies in Llama-3.1-8B Benchmark Evaluations Across Studies" (2026, 1.9%); "MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Contex…" (2025, 1.9%); "AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient…" (2026, 3.5%); "ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spa…" (2026, 85.6%). Preliminary analysis suggests: The extreme score variance likely stems from ReST-KV evaluating a fine-tuned or inference-optimized checkpoint with layer-wise reconstruction that artificially inflates retrieval accuracy, whereas AB-Sparse and MTraining report scores on the base model using strict sparse attention masks that severely degrade needle-i… Systematically evaluate which evaluation protocol factors (model configuration, inference setup, quantization, tokenization, few-shot count, metric interpretation, or data-split selection) best explain the observed spread; identify the highest-confidence explanation supported by each paper's stated methodology; and assess whether the highest-reported score is reproducible under the conditions described by the lowest-reporting paper.

Verification Level

Descriptive public verification status only; aggregate claim counts are public, but individual claim records are not exposed here.

Truth-Engine Gate Verdict

This record has not completed Gate 2 of the verification pipeline (a type-checked Lean4 proof for mathematical claims, or a sealed-sandbox reproduction for empirical claims). It is a literature synthesis only. VERIFIED requires an attached reproducible artifact (Lean4 proof source, or repro script and results) before this status can be set; it is not derived from review score or claim count.

Quality Tier

Descriptive public triage only; this tier does not alter current publication or DOI behavior.

Quality Dimensions

Automated triage signals derived from public fields; not human peer review or independent validation.

Correction Record

Public corrections are additive records. Current status does not claim the synthesis is error-free.

Provenance