WildDet3D brings 3D AI to any object

WildDet3D can detect objects in 3D from a single image, even when they are not part of a fixed list of categories. The model estimates where each object is, how large it is and how it is oriented, and lets you search for it with text, a point or a 2D box.

The technology, presented by the Allen Institute for AI, is designed for situations where a camera needs to understand physical space: a robot picking up packages, smart glasses identifying what is around you or an augmented reality app placing directions over a street.

From "what is it" to "where is it"

Most vision systems can recognize that there is a chair, a car or a person in a photo. The problem is determining how far away it is, how much space it occupies in the real world and which way it is facing using only one image.

WildDet3D tackles this problem with monocular 3D detection. In other words, it reconstructs the position and volume of objects from a single RGB image. You can type "fire extinguisher" to locate every fire extinguisher in the scene, tap an object in the image or provide a 2D box from another system.

The model also accepts very different types of images, such as a cropped photo taken with a phone, a wide-angle shot or footage from a robotic camera, without needing to be adjusted again for each camera.

When additional depth information is available, such as data from a LiDAR sensor, an RGB-D camera or a stereo system, WildDet3D can incorporate it to improve its estimates.

One model, several ways to ask it for something

The flexibility comes from how you tell it what to locate:

• Text: you enter a category, such as "chair", and the system finds its instances.
• Point: you point to a specific object and it returns its 3D box.
• 2D box: it receives a conventional detection and calculates its position and size in three dimensions.

It can also be combined with a vision-language model such as Molmo 2. That model interprets the user's request and passes the relevant region to WildDet3D to obtain its 3D location.

This connection can, for example, turn a 2D detector or tracking system into a 3D tracking tool. If a 2D box identifies an object in every frame of a video, WildDet3D can lift those detections into three-dimensional coordinates without having been specifically trained for tracking.

There is one important limitation: the full model still needs server-side processing or further optimization to run in real time directly on a device.

More than one million images for unusual objects

The project includes WildDet3D-Data, a dataset with more than one million images and 3.7 million verified 3D annotations. The material covers more than 13,000 object categories and includes more than 100,000 images reviewed by people.

To build it, the team generated possible 3D boxes from large collections of 2D images, including COCO, LVIS, Objects365 and V3Det. It then filtered and refined those proposals with several 3D estimation methods, vision-language models and human selection.

The goal is to avoid one of the common problems in 3D vision: training on a small number of categories and having the model work well only in specific settings, such as indoors or while driving. According to the team's evaluations, the new dataset helps it generalize to more than 700 categories in real-world situations.

Results in testing

On Omni3D, a benchmark that combines six indoor and outdoor datasets and 50 categories, WildDet3D achieves 34.2 AP using text queries. AP, or average precision, measures how closely the predicted 3D boxes match the actual positions and sizes. The result is 5.8 points higher than the previous best score cited by the team.

With reference 2D boxes, it reaches 36.4 AP. When sparse depth is added during testing, the figures rise to 41.6 AP with text and 45.8 AP with boxes.

The model was also tested without additional task-specific training on other datasets:

• 40.3 ODS on Argoverse 2, compared with 23.8 for the previous best result.
• 48.9 ODS on ScanNet, an improvement of 17.4 points.
• On new Argoverse 2 categories, 38.6 ODS versus 14.8.
• On new ScanNet categories, 45.8 versus 15.7.

ODS combines several aspects of detection, including precision, position, scale and orientation. The results matter particularly because they measure what happens when the object does not belong to the usual training categories.

On the team's WildDet3D-Bench, which covers more than 700 categories, the model reaches 22.6 AP with text queries after training on the additional data. On rare objects, it reaches 47.4 AP, compared with 2.4 for the comparison system cited by the team.

What changes in practice

For you, this means a camera can move from recognizing labels to building a more useful representation of space. An app could identify a specific table and calculate its dimensions; a robot could estimate where a box ends before grabbing it; and smart glasses could track an object's position as you move.

The project includes the model, the data, evaluation materials, an interactive demo and an iOS app. The latter uses the camera and LiDAR sensor on compatible devices to display 3D boxes over the image in augmented reality.

WildDet3D does not turn every camera into a perfect spatial perception system. Accuracy depends on the image, the depth available and the processing required. But its direction is clear: 3D detection is starting to move beyond closed environments and small lists of objects. The question now is whether these models can maintain that breadth with enough speed and reliability to run inside robots, phones and smart glasses, not just in demos or on servers.