How does the latency-accuracy trade-off of ExpertFlow's predictive caching compare to dense baselines and othe

Abstract

Abstract: Sparse Mixture-of-Experts (MoE) models can outperform dense large language models at similar computation by activating only a small set of experts per token. However, stacking many expert modules introduces substantial parameter memory, which makes MoE models difficult to deploy in memory-constrained environments such as single-GPU devices. Offloading alleviates this issue by storing inactive experts in CPU memory and loading them on demand, but existing methods remain limited: static caches disregard input-dependent routing, and methods that train separate models to predict expert usage ahead

Research Question

How does the latency-accuracy trade-off of ExpertFlow's predictive caching compare to dense baselines and other MoE routing strategies (e.g., Top-2, Hash Layers) on multi-object hallucination benchmarks (e.g., POPE, M-HalDetect) under throughput-constrained inference settings?

Verification Level

Descriptive public verification status only; aggregate claim counts are public, but individual claim records are not exposed here.

Quality Tier

Descriptive public triage only; this tier does not alter current publication or DOI behavior.

Quality Dimensions

Automated triage signals derived from public fields; not human peer review or independent validation.

Correction Record

Public corrections are additive records. Current status does not claim the synthesis is error-free.

Provenance