Falcon-H1-Arabic brings Arabic AI to 256K tokens

Falcon-H1-Arabic is TII’s new family of artificial intelligence models for understanding and generating Arabic, with versions containing 3 billion, 7 billion, and 34 billion parameters. The two largest models can process up to 256,000 tokens, enough to analyze several novels or hundreds of pages of technical material in a single conversation.

The launch aims to address common challenges in Arabic models: understanding dialects, staying on track through long texts, reasoning about mathematics, and working with specialized information. TII says all three models outperform other systems of similar size in its tests, although results depend on the benchmark and the configuration used.

A hybrid architecture for long texts

Falcon-H1-Arabic combines two ways of processing language within each model block. On one side, it uses Transformer attention, which helps it accurately connect distant words. On the other, it incorporates Mamba, an architecture based on state-space models that can handle long sequences more efficiently.

Both components work in parallel, and their representations are merged before the output is generated. The aim is to take advantage of Mamba’s speed on very long documents without losing attention’s ability to find important connections within the text.

The increase in context window is significant. Falcon-Arabic was limited to 32,000 tokens, while the new versions offer:

• 3B: 128,000 tokens, aimed at fast agents, low-resource devices, and systems handling many queries.
• 7B: 256,000 tokens, designed for production assistants, enterprise chat, and reasoning tasks.
• 34B: 256,000 tokens, intended for document analysis, research, and applications where errors carry a high cost.

A large context window does not, by itself, guarantee that the model will remember everything it receives. That is why TII says it trained the models to reduce the problem known as “lost in the middle”: ignoring relevant information located in the central part of a long document.

More real-world Arabic, not just formal Arabic

Training used around 300 billion tokens, with an almost even mix of Arabic, English, and other languages. This is intended to preserve capabilities for programming, solving STEM tasks, and working across languages without sacrificing Arabic-specific knowledge.

TII also increased the representation of dialects such as Egyptian, Levantine, Gulf, and Maghrebi Arabic. Arabic is not used in the same way across all countries, and a model trained mainly on Modern Standard Arabic can fail in everyday conversations or with regional expressions.

The company says it applied language-specific filters to detect issues with spelling, morphology, diacritics, and syntax. After initial training, the models went through supervised fine-tuning and preference-based optimization, two stages designed to improve instruction following, coherence, and response quality.

What results does it achieve

On the Open Arabic LLM Leaderboard, a test that measures Arabic comprehension capabilities, TII reports state-of-the-art results across all three sizes. The figures may differ slightly from the public leaderboard because the evaluation used vLLM instead of Accelerate, with differences usually below one point.

The reported results include:

• Falcon-H1-Arabic 3B: about 62% on OALL, around 82% on the native 3LM section, and approximately 50% on AraDice’s dialect evaluation.
• Falcon-H1-Arabic 7B: 71.7% on OALL, around 92% on native 3LM, and approximately 80% on ArabCulture.
• Falcon-H1-Arabic 34B: approximately 75% on OALL, around 96% on native 3LM, and approximately 80% on ArabCulture.

According to TII, the 34B model outperforms even considerably larger systems on OALL, such as Llama-3.3-70B. That does not mean it is better at every task or in every language, but it does suggest that an architecture and training data better adapted to Arabic can make up for a smaller parameter count in some evaluations.

What changes for you

The 3B version could work well for low-latency assistants, on-device applications, and agents that need to respond frequently without driving up costs. The 7B model targets general enterprise use: chatbots, document summarization, content generation, and information understanding.

The 34B model is designed for more demanding work, such as reviewing contracts, summarizing medical records, or analyzing extensive research. In these cases, its 256,000 tokens make it possible to submit large volumes of text without splitting them into as many parts and losing connections between them.

Even so, you should not treat the model as an automatic authority. TII warns that it may reproduce biases, make up information, and lose accuracy when used at the extreme limits of its context. In medical, legal, or financial settings, professional review will still be necessary.

Falcon-H1-Arabic shows that progress in AI does not depend only on building larger models. It also matters to train them on data that reflects how people speak and to design architectures that work well with the characteristics of each language. The next thing to watch is whether these improvements hold up beyond benchmarks, especially with dialects, sensitive documents, and real-world conversations lasting for extended periods.