New MoE Release from ai2 – EMO Model with Document-Level Routing
Use the new EMO model with 1B active/14B total trained on 1T tokens, leveraging document‑level routing for domain clustering.
Test EMO on your workloads, evaluate document‑level routing, and benchmark against existing LLMs.
Summary
ai2 has released the EMO Mixture‑of‑Experts model, featuring 1 B active parameters out of a total 14 B and trained on 1 T tokens. The model introduces document‑level routing, clustering experts around domains such as health and news rather than surface patterns. This routing allows the model to route entire documents to specialized experts, potentially improving domain‑specific performance. The release is available on HuggingFace under the ai2/emo collection. The model’s architecture supports large‑scale inference on modern GPUs. Users can experiment with the new routing scheme to evaluate its impact on prompt quality. The release marks a significant step in domain‑aware LLM design. The model is ready for integration into research pipelines and production workloads.
Key changes
- 1 B active, 14 B total, 1 T tokens
- Document‑level routing clusters experts by domain
- Experts focus on health, news, and similar domains
- Model released by ai2 and hosted on HuggingFace
- Supports large‑scale GPU inference
- Available as a new LLM release