LLaDA-MoE-7B-A1B
The first open-source sparse MoE diffusion language model — LLaDA-MoE-7B-A1B has 7B total parameters with only ~1.4B activated per token, pre-trained from scratch on ~20T tokens (Base / Instruct / Instruct-TD).
Overview
LLaDA-MoE is a new and upgraded series of the LLaDA diffusion language model family, released in the official LLaDA collection by inclusionAI. It is the first open-source Mixture-of-Experts (MoE) diffusion large language model, pre-trained from scratch on approximately 20 trillion tokens.
With 7B total parameters and only ~1.4B activated per token, LLaDA-MoE significantly reduces inference cost while outperforming open-source dense models of similar scale. It is particularly strong at code generation, advanced mathematical reasoning, and tool calling / agentic tasks.
Model Series
| Model | Description |
|---|---|
LLaDA-MoE-7B-A1B-Base | Base pre-trained model for research and secondary development |
LLaDA-MoE-7B-A1B-Instruct | Instruction-tuned model optimized for practical applications |
LLaDA-MoE-7B-A1B-Instruct-TD | Instruction-tuned variant with Trajectory Distillation for accelerated inference |
Highlights
- Sparse MoE diffusion: first open-source MoE architecture for discrete diffusion language modeling; only ~1B activated parameters per token in practice.
- Diffusion-style decoding: generates via block-wise masked diffusion with confidence-based remasking instead of next-token autoregression.
- 20T-token pre-training from scratch, with strong results on code and complex reasoning benchmarks and support for tool calling.
- Ecosystem: paired with the dInfer inference framework from inclusionAI.
