mmBERT Brings Multilingual AI to 1,833 Languages
mmBERT is an open AI model trained on more than 3 trillion tokens in 1,833 languages. Its creators say it outperforms models such as XLM-R and processes multilingual text faster, with contexts of up to 8,192 tokens.
A new AI model called mmBERT aims to improve how machines understand text in languages around the world, including those with limited available data. It was trained on more than 3 trillion tokens of text in 1,833 languages and, according to its creators, outperforms earlier multilingual models such as XLM-R in several tests.
The model was introduced by researchers at Johns Hopkins and is openly available on Hugging Face. Its goal is not to converse like a chatbot, but to understand and represent text for tasks such as search, classification, assisted translation, and question answering.
What mmBERT is and what it does
mmBERT is an encoder model. Instead of generating long texts one word at a time, it analyzes a complete sentence to capture its meaning. For example, it can help a search engine understand that “how to cancel a flight” and “I want to annul my airline reservation” express a similar intent.
This makes it useful for building tools such as:
- Search engines that work across multiple languages.
- Question-answering systems.
- Automatic classification of documents or messages.
- Detection of related content across different languages.
- Information retrieval and recommendation models.
There are two main versions: mmBERT-small, with 140 million parameters, and mmBERT-base, with 307 million. Parameters are the internal values the model adjusts during training to learn language patterns.
How it tries to learn languages with limited data
The challenge for multilingual models is uneven: the internet contains huge amounts of text in English, Spanish, or Russian, but far less in languages such as Tigrinya or Faroese. If every language is trained in the same way from day one, languages with more content can dominate the learning process.
mmBERT uses three phases to avoid this:
- Pretraining: processes 2.3 trillion tokens in 60 languages.
- Intermediate training: adds higher-quality data, expands the context to 8,192 tokens, and works with 110 languages.
- Final phase: adds all 1,833 languages during the final 100 billion tokens.
The amount of text hidden from the model also changes. It starts with a 30% masking rate, drops to 15%, and ends at 5%. This allows it to learn basic structures first and then focus on more precise relationships between words.
The authors say this strategy allows the model to quickly learn languages added at the end. In tests with Tigrinya and Faroese, mmBERT improved notably on question-answering tasks, despite seeing those languages only during the final phase.
Faster processing and longer texts
mmBERT inherits part of ModernBERT's architecture, but replaces its tokenizer, the system that divides text into units the model can process, with one based on Gemma 2. The change is intended to represent multilingual text more effectively.
It also uses Flash Attention 2 and techniques that eliminate unnecessary processing overhead. According to the published tests, the result is a model that is faster than other multilingual encoders and can process up to 8,192 tokens, compared with the usual limit of 512 tokens in earlier models.
For you, this could mean tools that analyze long documents in multiple languages with less memory and lower costs. A company could, for example, search for information in contracts written in Spanish, French, and German without having to maintain a separate model for each language.
It does not improve across the board
The results are not uniform. mmBERT outperforms earlier models in multilingual understanding, information retrieval, and English search, but its creators acknowledge that it remains roughly at the same level on tasks such as named entity recognition and part-of-speech tagging.
These tasks depend on precisely detecting word boundaries, something that may be affected by differences in the new tokenizer. In addition, the comparisons come from specific tests and do not guarantee that the model is the best option for every application.
mmBERT points to an important direction for open AI: it is not enough to add many languages to a model. You also have to decide when to add them, how much text to allocate to them, and how to prevent dominant languages from taking over the learning process. The next thing to watch is whether the community can turn these laboratory improvements into reliable systems for languages that have received little attention so far.