Falcon Perception brings AI to complex scenes

Falcon Perception is an open AI model that can find and crop objects in an image from detailed instructions, even when it has to read text, understand positions, or distinguish relationships between people and objects. Its main version has 0.6 billion parameters, a small scale compared with many current vision models.

The model was introduced by the Falcon Vision Team at the Technology Innovation Institute (TII) in Abu Dhabi. Its approach combines image and language understanding in a single architecture instead of separating the process into a vision encoder, a fusion module, and several specialized components.

One sequence for image and text

Falcon Perception converts image patches, instruction words, and task commands into a shared sequence. But it does not treat every element the same way:

• Image tokens can attend to the entire image to build a complete visual context.
• Text tokens are processed causally, using the image and previous words as context.
• Output instructions follow a fixed structure: coordinates, size, and mask.

This combination lets the model understand the scene as a visual system while responding step by step to a written request.

For each object, Falcon Perception first locates its center, then calculates its size, and finally generates a representation that is converted into a pixel mask. That order helps solve a common problem: knowing exactly which of several similar objects the instruction refers to.

For example, it does not simply locate any bottle. It can try to find the wine bottle with label 168, the black car to the right of the red car, or the person holding a helmet.

Better results when the instruction is difficult

On the open-vocabulary segmentation benchmark SA-Co, Falcon Perception achieves 68.0 Macro-F1, compared with 62.3 for SAM 3. Macro-F1 is a metric that summarizes average performance across several categories, giving each category equal weight.

The advantage grows when the task requires combining several clues. On the new PBench benchmark, designed to separate specific capabilities, the model outperforms SAM 3 in:

• Attributes: 63.6 versus 54.4.
• Identification guided by text read from the image: 38.0 versus 24.6.
• Spatial relationships: 53.5 versus 31.6.
• Relationships between objects or people: 49.1 versus 33.3.
• Dense scenes: 72.6 versus 58.4.

PBench also tests images with hundreds of objects. According to TII, Falcon Perception can segment those instances one after another, while the SAM 3 decoder can run out of available queries when there are more than about 200.

The result is not perfect. Its main weakness is presence calibration, meaning its ability to correctly decide when the requested object does not appear in the image. On that measure, Falcon Perception reaches an MCC of 0.64, compared with 0.82 for SAM 3. It can locate an object well when it is present, but it still needs to improve at saying when it is not there.

Falcon OCR also arrives

The announcement includes Falcon OCR, a separate model with 0.3 billion parameters for converting document images into structured text. It processes pages with multiple columns, tables, mathematical formulas, handwriting, and photographs taken in real-world conditions.

In the published tests, it achieves:

• 80.3% on olmOCR, just 1.7 points behind the leading system in that evaluation.
• 88.64 points on OmniDocBench, ahead of DeepSeek OCR v2, GPT 5.2, and Mistral OCR 3 in the comparison presented.
• 87.1% on multi-column documents and 90.3% on tables within olmOCR.

Its size is also intended to reduce the cost of serving the model. On an 80 GB A100, with high concurrency and vLLM, it reaches 2.9 images per second for the full layout detection and text recognition pipeline. TII says it is approximately three times smaller than OCR models in the 0.9 billion parameter class.

What this changes for you

The practical value lies in being able to give an image more precise instructions, rather than simply asking for a list of objects. An application could locate specific products on a shelf, review scanned documents, separate hundreds of elements in a photograph, or find a person based on the action they are performing.

The approach could also reduce deployment complexity: a shared backbone, lightweight outputs, and optimized attention for processing variable-length sequences. TII offers an inference stack based on PyTorch FlexAttention, integration with vLLM, and MLX support for devices with Apple chips.

Falcon Perception still needs to improve its ability to reject absent objects, and its results depend on resolution, sequence length, and the number of instances. But the announcement points in a clear direction: smaller models that simultaneously understand language, text inside images, geometry, and relationships without relying on an ever-longer chain of specialized modules.