EMO modularizes an AI model with 12.5% of its experts

AllenAI has introduced EMO, an artificial intelligence model that learns to organize its internal components by topic and capability. That means a specific task can run with just 12.5% of its experts without losing much performance.

The idea addresses a practical problem: large models often operate as single blocks, even when they do not need all their capabilities for every query. Asking for a mathematical formula, generating code, or summarizing a medical report does not require activating exactly the same knowledge.

EMO tries to separate those capabilities automatically. It does not start with categories defined by people, such as "mathematics," "biology," or "programming." It discovers them during training by observing the documents it processes.

A large model that can work in modules

EMO is a mixture-of-experts, or MoE, model. Instead of using the entire network for each word, a component called a router decides which small submodels, known as experts, should participate.

The model has 14 billion parameters in total, although it activates around 1 billion for each step. It consists of 128 experts, normally using eight at a time, and was trained on 1 trillion tokens, the units of text processed by a language model.

In theory, a system like this would load only the experts needed for a task. In practice, traditional MoE models do not usually work that way: different tokens in the same response can activate different experts, eventually using almost all of them.

The reason is that these experts do not always specialize in complete areas. They often end up grouping superficial patterns, such as punctuation marks, prepositions, or proper names. That does not help isolate a capability like mathematical reasoning.

The key is grouping complete documents

EMO introduces a restriction during training. The router must first choose a shared group of experts for each document, and then all its tokens are distributed within that group.

A health-related article, for example, will tend to use a set of experts related to health and medicine. A text about code may rely on another group. The model is not given labels telling it what topic each document covers: the structure emerges by observing the data.

To prevent all documents from always using the same experts, EMO balances the load at a global level. During training, it also varies the size of the groups, so the model is not limited to a single configuration when it is time to select experts.

How much performance it preserves

The published results indicate that EMO maintains performance similar to that of a conventional MoE when all its experts are used. The difference appears when some of them are removed.

• With 25% of the experts, it loses around one percentage point of performance on the evaluated benchmarks.
• With 12.5%, the average drop is around three percentage points.
• A conventional MoE with the same architecture degrades much more when its set of experts is reduced.

To choose the right experts, it is enough to analyze a few examples from the task. AllenAI says that even a single example with demonstrations can identify a module that is nearly as useful as one selected using a complete validation set.

This does not mean you can simply delete 87.5% of the model. First, you need to select which experts fit the task, and the result depends on that selection being correct. But the process requires far less information and memory than always running the complete model.

What changes for you

If these results hold up in larger models, an application could adapt its resource use to what you are doing:

• A coding assistant would load mainly modules related to code.
• A medical tool could work with a specialized subset.
• A general-purpose system would continue using all the experts when it needed to answer questions on varied topics.

The advantage would be twofold: less memory to host the model and less computation for each query. It would also be easier to update or study a specific capability without modifying the entire network.

AllenAI has published the EMO model, a conventional MoE version trained on the same data, along with the training code. That material will make it possible to test whether the modularity holds outside the original experiments and how it works with other techniques for pruning experts.

EMO does not yet solve how to combine modules, update them without affecting the rest, or ensure that each one represents a stable capability. But it points in a clear direction: instead of treating large models as indivisible blocks, the goal is to build systems that can divide themselves and use only what they need.