AILaboxArchitectureModel structures: Transformers, attention and mixture-of-experts
混合专家（MoE）
An architecture that splits a model into "expert" subnetworks, activating only a few per token.
MoE routes each token through a few experts, getting large capacity at lower compute cost. Total parameters (e.g. 671B) far exceed activated parameters (e.g. 37B), a cost-effective choice for models like DeepSeek-V3.