Reproducibility Meta-Analysis of Divergent Llama-3 Belebele Performance Attributed to Model Checkpoints and Evaluation Protocols

Abstract

Abstract: User-generated content (UGC) on social media can act as a key source of information for emergency responders in crisis situations. However, due to the volume concerned, computational techniques are needed to effectively filter and prioritise this content as it arises during emerging events. In the literature, these techniques are trained using annotated content from previous crises. In this paper, we investigate how this prior knowledge can be best leveraged for new crises by examining the extent to which crisis events of a similar type are more suitable for adaptation to new events (cross-dom

Research Question

Reproducibility meta-analysis: 2 independent publications report divergent Llama-3 performance on Belebele with a 54.2 percentage-point spread (range 35.8%–90.0%). Source papers: "Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptation" (2025, 35.8%); "AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs" (2024, 90.0%). Preliminary analysis suggests: The discrepancy likely stems from the use of different model checkpoints, where the 90.0% score reflects a fine-tuned or dialect-specific variant optimized for Arabic capabilities in AraDiCE, while the 35.8% result corresponds to a base Llama-3 model evaluated under strict zero-shot cross-lingual conditions without ta… Systematically evaluate which evaluation protocol factors (model configuration, inference setup, quantization, tokenization, few-shot count, metric interpretation, or data-split selection) best explain the observed spread; identify the highest-confidence explanation supported by each paper's stated methodology; and assess whether the highest-reported score is reproducible under the conditions described by the lowest-reporting paper.

Verification Level

Descriptive public verification status only; aggregate claim counts are public, but individual claim records are not exposed here.

Truth-Engine Gate Verdict

This record has passed Gate 2: a Lean4 proof source type-checks, or a sealed-sandbox run reproduced the reported results within the stated tolerance. A reproducible artifact (proof source or repro script and results) is attached to this record. VERIFIED requires an attached reproducible artifact (Lean4 proof source, or repro script and results) before this status can be set; it is not derived from review score or claim count.

Quality Tier

Descriptive public triage only; this tier does not alter current publication or DOI behavior.

Quality Dimensions

Automated triage signals derived from public fields; not human peer review or independent validation.

Correction Record

Public corrections are additive records. Current status does not claim the synthesis is error-free.

Provenance