Ai2 introduces Bolmo, AI models that read bytes

Ai2 has introduced Bolmo, a family of language models that processes text directly as bytes, the basic units of digital files, instead of splitting it into word fragments. The approach aims to improve how models handle characters, spelling errors, symbols, and multilingual text without giving up the performance of current models.

Language models typically work with tokens: word fragments such as inter, national, or ization. This system is efficient, but it can struggle with uncommon words, spaces, mixed languages, or tasks that depend on individual characters, such as checking a password or generating code precisely.

Bolmo aims to solve that problem without training a model entirely from scratch. Ai2 used its open Olmo 3 models as a foundation and adapted them to work with bytes. The result is two models, Bolmo 7B and Bolmo 1B, which the organization presents as fully open byte models that are competitive with token-based alternatives.

How Bolmo works

Each byte in a text first passes through a local encoder that analyzes its nearby context. A predictor then decides where to group those bytes into variable-length blocks called patches. These blocks are sent to the main transformer, which retains the overall architecture of Olmo 3.

At the end, the model breaks the information back down into bytes to predict the next character and decide the next boundary. The important difference is that the boundaries are not fixed: Bolmo can group more or fewer bytes depending on the information they contain.

This combines two advantages. The model retains the capabilities of a large transformer trained with proven data and techniques, but it can also examine text at a level of detail that tokens do not always provide.

More character-level precision, without starting from scratch

Training a byte model from scratch is usually expensive because it has to relearn much of what token models already know. Ai2 chose a different path.

First, it froze the Olmo 3 transformer and trained the new components with 9.8 billion tokens, equivalent to approximately 43 billion bytes. It then unfroze the entire model and added another 39.3 billion tokens, around 173 billion bytes, so Bolmo could make better use of character-level information.

In Ai2's tests, Bolmo 7B came close to Olmo 3 7B on general evaluations covering math, science, question answering, code, and general knowledge. It performed clearly better on character-focused tests such as CUTE and EXECUTE.

Across the combined set of character-level tasks, Bolmo 7B improved accuracy by almost 20 points over Olmo 3. Compared with other byte models of a similar size, such as BLT 7B, TFree-Hat 7B, and EvaByte 6.5B, Ai2 says it achieved the best overall result in code, math, multiple-choice questions, and character understanding, although it scored slightly below TFree-Hat 7B on GenQA.

Is it slower?

Working with bytes can mean processing many more units than working with split words. Bolmo tries to offset that by grouping bytes dynamically.

In measurements shared by Ai2, Bolmo reaches around 125 bytes per second, compared with about 150 bytes per second for its token-based model with comparable compression. Speed can increase if the model groups more bytes into each block, although that requires adjusting the balance between speed and detail.

This compression works like an adjustable control. In a traditional model, expanding the vocabulary can make the layer that selects the next token heavier. Bolmo can seek greater efficiency by increasing the average number of bytes per block, without having to redesign the entire vocabulary.

What this changes for you

The advantage is not that Bolmo will automatically replace today's chatbots. Its value lies in cases where small details matter a great deal:

• Checking spelling, formats, and text strings.
• Working with code, symbols, and uncommon characters.
• Processing languages and writing systems with less representation in standard vocabularies.
• Handling misspelled, mixed, or unusually structured text.
• Adjusting speed and level of detail according to the task.

Ai2 also showed that Bolmo can inherit capabilities from an Olmo 3 version trained to follow instructions without repeating the entire training process. Using a weight-merging technique, the model went from 31.1% to 67.4% on IFEval, nearly matching the 66.9% achieved by the original instruction-trained Olmo 3 version.

That opens up a relevant possibility for open models: adapt a token-based model to bytes and then reuse some of its tuning, adapters, and improvements without rebuilding the entire ecosystem from scratch. Ai2 warns that this compatibility is not guaranteed for every model family.

Bolmo is still a research proposal, not proof that bytes are always better than tokens. What it offers is a practical way to test them with already capable open models. The next step will be finding out whether this approach maintains its advantages as it scales to larger models, other languages, and specialized applications.