Systematic Analysis of Protocol Factors Driving Extreme Rouge-L Variance in GPT-3.5 Across Divergent Evaluation Domains

Abstract

Abstract: Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions:

Research Question

Reproducibility meta-analysis: 4 independent publications report divergent GPT-3.5 performance on Rouge-L with a 78.2 percentage-point spread (range 1.8%–80.0%). Source papers: "ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using…" (2025, 1.8%); "Automated Literature Review Using NLP Techniques and LLM-Based Retrieval-Augmen…" (2024, 18.1%); "A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot" (2023, 18.9%); "Comparison of Open-Source and Proprietary LLMs for Machine Reading Comprehensio…" (2024, 80.0%). Preliminary analysis suggests: The extreme variance likely stems from evaluating fundamentally different tasks under the same "Rouge-L" label, where the 80.0% score reflects a standard text-based Machine Reading Comprehension benchmark while the <20% scores originate from cross-modal video-to-text generation or specialized code evaluation domains w… Systematically evaluate which evaluation protocol factors (model configuration, inference setup, quantization, tokenization, few-shot count, metric interpretation, or data-split selection) best explain the observed spread; identify the highest-confidence explanation supported by each paper's stated methodology; and assess whether the highest-reported score is reproducible under the conditions described by the lowest-reporting paper.

Verification Level

Descriptive public verification status only; aggregate claim counts are public, but individual claim records are not exposed here.

Truth-Engine Gate Verdict

This record has not completed Gate 2 of the verification pipeline (a type-checked Lean4 proof for mathematical claims, or a sealed-sandbox reproduction for empirical claims). It is a literature synthesis only. VERIFIED requires an attached reproducible artifact (Lean4 proof source, or repro script and results) before this status can be set; it is not derived from review score or claim count.

Quality Tier

Descriptive public triage only; this tier does not alter current publication or DOI behavior.

Quality Dimensions

Automated triage signals derived from public fields; not human peer review or independent validation.

Correction Record

Public corrections are additive records. Current status does not claim the synthesis is error-free.

Provenance