MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

Abstract

Abstract: Recent advances demonstrate that scaling Large Vision-Language Models (LVLMs) effectively improves downstream task performances. However, existing scaling methods enable all model parameters to be active for each token in the calculation, which brings massive training and inferring costs. In this work, we propose a simple yet effective training strategy MoE-Tuning for LVLMs. This strategy innovatively addresses the common issue of performance degradation in multi-modal sparsity learning, consequently constructing a sparse model with an outrageous number of parameters but a constant computation

Research Question

How does SMoES's routing strategy compare to dense models and modality-agnostic MoE baselines in terms of Top-1 accuracy on the MMMU benchmark under visual vs. textual adversarial perturbations across model scales (e.g., 7B, 13B)?

Verification Level

Descriptive public verification status only; aggregate claim counts are public, but individual claim records are not exposed here.

Quality Tier

Descriptive public triage only; this tier does not alter current publication or DOI behavior.

Quality Dimensions

Automated triage signals derived from public fields; not human peer review or independent validation.

Correction Record

Public corrections are additive records. Current status does not claim the synthesis is error-free.

Provenance