Does the accuracy gap on long-context multimodal benchmarks (e.g., Video-MME, Needle-in-a-Haystack) between Mo

Abstract

Abstract: We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in particular textit\the ability to utilize information at arbitrary input locations\, is a capability that is mostly already acquired through large-scale pretraining, and that this capability can be readily extended to contexts substantially longer than seen during training(e.g., 4K to 128K) through lightweight continual pretraining on appropriate data mixture. We investigate the textit\quantity\ and textit\quality\ of

Research Question

Does the accuracy gap on long-context multimodal benchmarks (e.g., Video-MME, Needle-in-a-Haystack) between MoE and dense models widen or narrow as context length increases from 128K to 10M tokens?

Verification Level

Descriptive public verification status only; aggregate claim counts are public, but individual claim records are not exposed here.

Quality Tier

Descriptive public triage only; this tier does not alter current publication or DOI behavior.

Quality Dimensions

Automated triage signals derived from public fields; not human peer review or independent validation.

Correction Record

Public corrections are additive records. Current status does not claim the synthesis is error-free.

Provenance