Systematic Evaluation of Protocol Factors Driving Divergent mBERT Performance on Maltese Part-of-Speech Tagging

Abstract

Abstract: Abstract Background To evaluate binary classifications and their confusion matrices, scientific researchers can employ several statistical rates, accordingly to the goal of the experiment they are investigating. Despite being a crucial issue in machine learning, no widespread consensus has been reached on a unified elective chosen measure yet. Accuracy and F 1 score computed on confusion matrices have been (and still are) among the most popular adopted metrics in binary classification tasks. However, these statistical measures can dangerously show overoptimistic inflated results, especially on

Research Question

Reproducibility meta-analysis: 2 independent publications report divergent mBERT performance on Pos with a 74.6 percentage-point spread (range 14.3%–88.9%). Source papers: "XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-ling…" (2020, 14.3%); "Feature Aggregation in Zero-Shot Cross-Lingual Transfer Using Multilingual BERT" (2022, 70.3%); "Cross-Lingual Transfer from Related Languages: Treating Low-Resource Maltese as…" (2024, 88.9%). Preliminary analysis suggests: The most likely explanation is a combination of evaluation protocol differences and model checkpoint variations. The 88.9% score likely reflects a fine-tuned or adapted mBERT model with task-specific adjustments (e.g., back-translation or continued pre-training on Maltese), while the 14.3% score may stem from a zero-s… Systematically evaluate which evaluation protocol factors (model configuration, inference setup, quantization, tokenization, few-shot count, metric interpretation, or data-split selection) best explain the observed spread; identify the highest-confidence explanation supported by each paper's stated methodology; and assess whether the highest-reported score is reproducible under the conditions described by the lowest-reporting paper.

Verification Level

Descriptive public verification status only; aggregate claim counts are public, but individual claim records are not exposed here.

Truth-Engine Gate Verdict

This record has not completed Gate 2 of the verification pipeline (a type-checked Lean4 proof for mathematical claims, or a sealed-sandbox reproduction for empirical claims). It is a literature synthesis only. VERIFIED requires an attached reproducible artifact (Lean4 proof source, or repro script and results) before this status can be set; it is not derived from review score or claim count.

Quality Tier

Descriptive public triage only; this tier does not alter current publication or DOI behavior.

Quality Dimensions

Automated triage signals derived from public fields; not human peer review or independent validation.

Correction Record

Public corrections are additive records. Current status does not claim the synthesis is error-free.

Provenance