What is the computational overhead of implementing expert bridging versus full fine-tuning in terms of inferen

Abstract

Abstract: Fine-tuning Large Language Models (LLMs) is a common practice to adapt pre-trained models for specific applications. While methods like LoRA have effectively addressed GPU memory constraints during fine-tuning, their performance often falls short, especially in multi-task scenarios. In contrast, Mixture-of-Expert (MoE) models, such as Mixtral 8x7B, demonstrate remarkable performance in multi-task learning scenarios while maintaining a reduced parameter count. However, the resource requirements of these MoEs remain challenging, particularly for consumer-grade GPUs with less than 24GB memory. To

Research Question

What is the computational overhead of implementing expert bridging versus full fine-tuning in terms of inference latency and memory usage across different model scales from 110M to 175B parameters?

Verification Level

Descriptive public verification status only; aggregate claim counts are public, but individual claim records are not exposed here.

Quality Tier

Descriptive public triage only; this tier does not alter current publication or DOI behavior.

Quality Dimensions

Automated triage signals derived from public fields; not human peer review or independent validation.

Correction Record

Public corrections are additive records. Current status does not claim the synthesis is error-free.

Provenance