Impact of Layer-wise KV Cache Reconstruction on Artificially Inflated Needle-in-a-Haystack Scores in Ultra-Long Context Tasks

Abstract

Abstract: Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70\% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical semantic relationships between tokens, resulting in fragmented context and degraded performance. We introduce ChunkKV, which fundamentally reimagines KV cache compression by treating semantic chunks - rather than isolated tokens - as basic compression units. This approach preserves complete linguisti

Research Question

To what extent does layer-wise KV cache reconstruction in methods like ReST-KV artificially inflate needle-in-a-haystack scores relative to standard eviction policies on ultra-long context tasks?

Verification Level

Descriptive public verification status only; aggregate claim counts are public, but individual claim records are not exposed here.

Truth-Engine Gate Verdict

This record has passed Gate 2: a Lean4 proof source type-checks, or a sealed-sandbox run reproduced the reported results within the stated tolerance. A reproducible artifact (proof source or repro script and results) is attached to this record. VERIFIED requires an attached reproducible artifact (Lean4 proof source, or repro script and results) before this status can be set; it is not derived from review score or claim count.

Quality Tier

Descriptive public triage only; this tier does not alter current publication or DOI behavior.

Quality Dimensions

Automated triage signals derived from public fields; not human peer review or independent validation.

Correction Record

Public corrections are additive records. Current status does not claim the synthesis is error-free.

Provenance