NVIDIA launches Nemotron 3 Nano 4B for local AI

NVIDIA has introduced Nemotron 3 Nano 4B, an artificial intelligence model with 4 billion parameters designed to run on computers, graphics cards, and small devices without relying continuously on the cloud.

Its size makes it suitable for systems such as GeForce RTX GPUs, Jetson Orin Nano and Jetson Thor modules, and the DGX Spark system. That opens the door to conversational assistants, gaming tools, and robots that respond locally, with lower latency and greater control over data.

What this model offers

Nemotron 3 Nano 4B combines a hybrid Mamba-Transformer architecture. In simple terms, it mixes two ways of processing text to maintain solid performance while using less memory.

NVIDIA says the model achieves benchmark-leading results in its category across several tests:

• Instruction following, meaning understanding and carrying out what you ask.
• Tool use and agent behavior, such as calling a function or completing multiple steps.
• Intelligence for tactical video games.
• Peak video memory usage.
• Time to generate the first response in tasks with long contexts.

The efficiency comparisons were conducted on an RTX 4070 using llama.cpp and versions compressed with Q4_K_M, a 4-bit quantization technique that reduces the model's size.

Smaller than its source model

NVIDIA did not train the 4-billion-parameter model from scratch. It derived it from Nemotron Nano 9B v2, a 9-billion-parameter model, through structured pruning and distillation.

Pruning removes parts of the model that contribute less to its operation. Distillation uses the larger model as a reference so the smaller one can recover as much of its capability as possible. The result goes from 56 to 42 layers and reduces both the internal dimensions and the number of processing heads.

NVIDIA then trained it on data covering reasoning, programming, mathematics, science, conversation, and agent tasks. It also applied reinforcement learning to improve instruction following and tool use.

Although it can reason, the model is tuned to solve many tasks without having to display an explicit chain of thought. It also includes specific training to reinforce safe behavior.

What changes when you run it on a device

The main advantage is that certain AI functions can run directly on the device. For example, an assistant installed on a laptop or robot could process a command without sending every message to an external server.

That can mean:

• Lower latency, because the information does not have to travel to a data center.
• More privacy, especially for conversations or sensor data.
• Lower usage costs, by reducing reliance on pay-per-use services.
• Offline operation, as long as the device has the model installed.

On an 8 GB Jetson Orin Nano, NVIDIA says the Q4_K_M version reaches 18 tokens per second with llama.cpp, up to twice the performance of Nemotron Nano 9B v2. On DGX Spark and Jetson Thor, the FP8 version delivers up to 1.8 times the latency and processing performance of the original BF16 version, according to the company's tests.

The FP8 version keeps some layers at higher precision to limit quality loss. NVIDIA says that both this variant and the GGUF version recover 100% of the median accuracy of the BF16 version on the selected benchmarks.

Available for different environments

The model is released as open software in three main formats:

• BF16, with higher precision and higher memory use.
• FP8, aimed at balancing quality and efficiency.
• Q4_K_M in GGUF format, designed to run with llama.cpp on more limited hardware.

It is also compatible with Transformers, vLLM, and TensorRT-LLM, in addition to llama.cpp. That makes it possible to integrate it into development projects as well as applications installed directly on a PC, GPU, or edge device.

The news matters because it brings specialized assistants to devices that do not have constant access to large data centers. The next point to watch will be how it performs outside benchmarks: in long conversations, real-world tasks, and devices with thermal or memory constraints. Having 4 billion parameters makes it easier to install, but that alone does not guarantee it will be the best option for every use case.