EMO creates AI modules that work independently
Ai2 has introduced EMO, an MoE model that learns to organize its experts by topic from its training documents. It can retain nearly all its performance using only 12.5% of its experts, pointing toward AI models that are cheaper, more adaptable, and easier to deploy.
Ai2 has introduced EMO, an artificial intelligence model whose experts learn to organize themselves by topic without anyone telling them in advance which ones should handle math, coding, or medicine. The result lets you use only part of the model without giving up much performance.
EMO is a mixture-of-experts, or MoE, model. Instead of activating the entire network for every word, an internal system called a router decides which small modules, known as experts, should process it.
The idea sounds simple, but current MoE models have a problem: even though each word activates only a few experts, a complete task can end up using almost all of them. Those experts also tend to specialize in superficial details, such as punctuation marks, prepositions, or proper names, instead of learning useful capabilities like programming or mathematical reasoning.
Modules that emerge from the data
EMO tries to solve this without imposing human-defined categories during training. Its technique starts from a basic observation: words in the same document usually belong to the same topic.
During training, the tokens in each document, meaning the units the model uses to divide up text, are required to choose their experts from a shared group. The router decides which group that is by looking at the preferences of all the tokens in the document.
As a result, a health article tends to activate one set of health-related experts, while a political news story activates another. The groups are not defined with manual labels. They emerge from the content used to train the model.
Training also uses different group sizes and a global balancing system. This prevents the model from concentrating all the work in a few experts and allows groups of different sizes to be selected when the model is used.
What results does it achieve
EMO has 14 billion parameters in total, although it activates around 1 billion for each input. It consists of 128 experts, of which 8 are normally activated per token, and was trained on 1 trillion tokens.
When all its experts are used, EMO matches the performance of a conventional MoE model with the same architecture and trained on the same data. The difference appears when some of them are removed:
- With 25% of the experts, it loses around 1 percentage point of performance.
- With 12.5%, equivalent to 16 of the 128 experts, the drop is around 3 percentage points.
- The conventional MoE model degrades much more when its set of experts is reduced and, in some cases, approaches random performance.
To choose the right experts for a task, you only need to analyze a few examples. Ai2 says that even a single example with demonstrations can identify a group as effectively as one selected using a full validation set.
What changes for you
If this technique carries over to larger models, you could run only the specialized part you need instead of loading the entire network. A company could use one module to review contracts, another to write code, and another to analyze medical data, without keeping all the parameters active at the same time.
That would reduce memory use and the cost of adapting or deploying large models. It would also make it easier to update them in parts, although EMO has not yet shown that new knowledge can be added to one module without affecting the rest of the system.
The model's internal analyses point to a clear difference. Its expert groups are associated with topics such as health, news, United States politics, film, and music. In the conventional model, the groups are organized around linguistic patterns such as definite articles, copular verbs, or proper names.
Ai2 has published the EMO model, a conventional MoE reference model, and the training code. The next step will be learning how to combine modules, update them without breaking the full model, and use them to better understand how an AI makes decisions. EMO does not yet turn large models into interchangeable components, but it shows a concrete path for moving them beyond single, indivisible blocks.