Reproducibility Meta-Analysis of Divergent Qwen3 MATH Benchmarks: Evaluating Protocol Factors Behind a 75-Point Performance Spread

Abstract

Abstract: We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5\% and 17.0\%, respectively, which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully connected layers with a final 1000-way softmax. To make training faster, we used non-saturat

Research Question

Reproducibility meta-analysis: 2 independent publications report divergent Qwen3 performance on MATH with a 75.0 percentage-point spread (range 0.0%–75.0%). Source papers: "SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Mult…" (2025, 0.0%); "DiffCoT: Diffusion-styled Chain-of-Thought Reasoning in LLMs" (2026, 75.0%). Preliminary analysis suggests: The extreme discrepancy likely stems from SPIRAL evaluating a base pre-training checkpoint without mathematical instruction tuning or specific chain-of-thought prompting, whereas DiffCoT reports results on a model fine-tuned with its specialized diffusion-style reasoning framework. Additionally, the 0.0% score suggest… Systematically evaluate which evaluation protocol factors (model configuration, inference setup, quantization, tokenization, few-shot count, metric interpretation, or data-split selection) best explain the observed spread; identify the highest-confidence explanation supported by each paper's stated methodology; and assess whether the highest-reported score is reproducible under the conditions described by the lowest-reporting paper.

Verification Level

Descriptive public verification status only; aggregate claim counts are public, but individual claim records are not exposed here.

Truth-Engine Gate Verdict

This record has not completed Gate 2 of the verification pipeline (a type-checked Lean4 proof for mathematical claims, or a sealed-sandbox reproduction for empirical claims). It is a literature synthesis only. VERIFIED requires an attached reproducible artifact (Lean4 proof source, or repro script and results) before this status can be set; it is not derived from review score or claim count.

Quality Tier

Descriptive public triage only; this tier does not alter current publication or DOI behavior.

Quality Dimensions

Automated triage signals derived from public fields; not human peer review or independent validation.

Correction Record

Public corrections are additive records. Current status does not claim the synthesis is error-free.

Provenance