AI2 presents Olmo Hybrid, a 7B hybrid model

AI2 has introduced Olmo Hybrid, an open language model with 7 billion parameters that combines two different ways of processing text. In controlled tests, it matches Olmo 3 7B on MMLU using 49% fewer tokens, equivalent to training it on roughly half the data.

The result points to a possible major improvement in the cost of creating AI models, although it does not mean Olmo Hybrid is better at absolutely every task. Its advantage appears mainly when scaling training and working with very long texts.

Combining two architectures

Current models are usually based on transformers. Their attention system can precisely locate any earlier piece of text, which is useful for answering questions about documents or following complex instructions. The problem is that the cost of this attention rises quickly as the context gets longer.

A text that is twice as long may require roughly four times as many attention operations. That makes maintaining long conversations or analyzing lengthy documents more expensive.

Linear recurrent neural networks take a different approach. They process information progressively and maintain an internal state that updates with each word. This makes them well suited to tracking a changing situation, such as a sports score or the state of a chess game, and their cost grows linearly with the length of the text.

But that internal state compresses earlier information. If the model needs to retrieve an exact detail mentioned many pages earlier, it may have more difficulty.

Olmo Hybrid combines both systems: it alternates transformer attention layers with Gated DeltaNet layers, a modern type of linear recurrent network that can also be trained in parallel. The architecture uses a pattern of three DeltaNet layers for every attention layer. In practice, it replaces 75% of the attention-based mixing layers while retaining enough attention layers to retrieve specific information.

Less data to reach the same result

AI2 compared Olmo Hybrid with Olmo 3 7B, keeping the model size and training speed as consistent as possible. Both models have around 7 billion parameters and were trained at comparable rates.

The main differences appeared in the amount of data needed to reach a given capability:

• On MMLU, a benchmark for general knowledge and reasoning, Olmo Hybrid matched Olmo 3 using 49% fewer tokens.
• In an evaluation based on a Common Crawl sample, it reached the same level using 35% fewer tokens.
• During full training, both models used 6 trillion tokens.

Using fewer tokens to achieve the same result means less training time and, under similar conditions, less compute consumption. It also creates the possibility of using the same budget to train a more capable model.

The model was trained on 512 GPUs, first with NVIDIA H100s and later with NVIDIA HGX B200s. AI2 says the improvements mainly come from the hybrid architecture, not from training faster or using a larger model.

An advantage with long contexts

The difference becomes especially visible when the model has to handle long inputs. After adapting it for long contexts, Olmo Hybrid outperformed Olmo 3 7B on RULER, a test designed to measure the ability to work with large amounts of text.

At 64,000 tokens of context, the results were as follows:

• Olmo Hybrid with DRoPE: 85.0 points.
• Olmo Hybrid with YaRN: 76.9 points.
• Olmo 3 7B with YaRN: 70.9 points.

At 4,000 tokens, Olmo Hybrid was slightly behind. From 8,000 tokens onward, it took the lead, and the difference increased as the context length grew.

For you, this could mean more affordable models for analyzing contracts, records, code, or long conversations. The benefit is not only that the system can accept more text, but also that it can do so with a cost that grows more slowly.

It does not win every test

The results have some nuance. At the end of pretraining, Olmo Hybrid performed better on some math and science tests but fell slightly behind on programming and general questions.

After an additional training phase, those differences closed, and the model outperformed Olmo 3 across all the main domains evaluated. It also improved on BBH and MMLU Pro, although it recorded small regressions on LBPP and DM Math.

AI2 also argues that the hybrid model is more expressive than either a pure transformer or a linear recurrent network on its own. In this context, expressiveness means that it can represent a broader range of useful operations and patterns during learning.

Scaling analyses suggest that the token savings could increase with model size: from approximately 1.3 times in models with 1 billion parameters to 1.9 times in models with 70 billion, according to its estimates. These projections come from scaling curves and are not results already demonstrated with a 70-billion-parameter model.

Olmo Hybrid does not show that transformers are going to disappear. It does provide a controlled comparison that strengthens a direction already being explored by other projects: combining precise memory with an internal state that is cheaper to update. The next step will be to determine whether these advantages hold in larger models, generative tasks, and real-world use with long contexts.