Reproducibility Meta-Analysis of Divergent Llama-3 Longbench Performance Across Edge and Server Inference Protocols

Abstract

Abstract: In this report, we introduce Qwen2.5, a comprehensive series of large language models (LLMs) designed to meet diverse needs. Compared to previous iterations, Qwen 2.5 has been significantly improved during both the pre-training and post-training stages. In terms of pre-training, we have scaled the high-quality pre-training datasets from the previous 7 trillion tokens to 18 trillion tokens. This provides a strong foundation for common sense, expert knowledge, and reasoning capabilities. In terms of post-training, we implement intricate supervised finetuning with over 1 million samples, as well

Research Question

Reproducibility meta-analysis: 3 independent publications report divergent Llama-3 performance on Longbench with a 94.2 percentage-point spread (range 5.8%–100.0%). Source papers: "Understanding the Performance and Power of LLM Inferencing on Edge Accelerators" (2025, 5.8%); "SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thr…" (2024, 49.4%); "ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference" (2024, 100.0%). Preliminary analysis suggests: The extreme variance likely stems from the 5.8% score evaluating a quantized or distilled edge-optimized checkpoint under strict memory constraints that cripple long-context retention, whereas the 100% and 49.4% scores reflect full-precision server-grade models using different attention mechanisms (ShadowKV vs. SageAt… Systematically evaluate which evaluation protocol factors (model configuration, inference setup, quantization, tokenization, few-shot count, metric interpretation, or data-split selection) best explain the observed spread; identify the highest-confidence explanation supported by each paper's stated methodology; and assess whether the highest-reported score is reproducible under the conditions described by the lowest-reporting paper.

Verification Level

Descriptive public verification status only; aggregate claim counts are public, but individual claim records are not exposed here.

Truth-Engine Gate Verdict

This record has not completed Gate 2 of the verification pipeline (a type-checked Lean4 proof for mathematical claims, or a sealed-sandbox reproduction for empirical claims). It is a literature synthesis only. VERIFIED requires an attached reproducible artifact (Lean4 proof source, or repro script and results) before this status can be set; it is not derived from review score or claim count.

Quality Tier

Descriptive public triage only; this tier does not alter current publication or DOI behavior.

Quality Dimensions

Automated triage signals derived from public fields; not human peer review or independent validation.

Correction Record

Public corrections are additive records. Current status does not claim the synthesis is error-free.

Provenance