⚡ Nemotron 3 Nano 4B: NVIDIA's Edge Model on Your RTX Card
A 4B hybrid Mamba-Transformer model NVIDIA built for Jetson and RTX hardware, not data centers. Architecture, benchmarks, license, and the run command.
Derek Holt · Local AI & Hardware Writer
· 4 min read
The headline spec on Nemotron 3 Nano 4B isn't the parameter count — 4B is a crowded weight class. It's the layer breakdown: 21 Mamba-2 layers, 17 MLP layers, and only 4 attention layers, out of 42 total. Attention layers are the ones that get expensive as context grows. Trading most of them out for Mamba is the whole reason this model targets a Jetson Orin Nano instead of a rack.
By the numbers
| Spec | Value |
|---|---|
| Parameters | 3.97B |
| Architecture | Hybrid: 21 Mamba-2 / 4 Attention / 17 MLP layers (42 total) |
| Embedding dimension | 3,136 |
| MATH500 (reasoning mode) | 95.4% |
| Jetson Orin Nano throughput | ~18 tok/s, up to 2× Nemotron Nano 9B v2 at Q4_K_M |
| License | NVIDIA Nemotron Open Model License (commercial-permissive) |
| Quant formats | BF16, FP8 (100% median accuracy recovery vs BF16), Q4_K_M GGUF |
| Deployment targets | Jetson Thor/Orin Nano, DGX Spark, RTX GPUs |
Why the Mamba-heavy layer split actually matters
Standard transformer attention gets more expensive as the sequence gets longer — every new token has to weigh itself against every prior token. Mamba-2 layers process sequence information through a state-space mechanism that scales close to linearly instead. NVIDIA's design keeps just 4 attention layers in the whole 42-layer stack, using them where they earn their cost, and lets Mamba carry the rest. The practical result: on a Jetson Orin Nano at Q4_K_M quantization, NVIDIA reports roughly 2× the throughput of the 9B parent model it was distilled from — not just "smaller and therefore faster," but faster per-parameter because of what the parameters are doing.
It was built for Jetson first, RTX second — and that's fine
Be clear-eyed about what this model is for: NVIDIA's primary target is edge deployment — robotics, embedded agents, on-device assistants running on Jetson Thor and Orin Nano hardware where every watt and every gigabyte of RAM is contested. RTX desktop cards are a secondary, easier target by comparison. If you're running this on a 16GB RTX 4080 the way we do for our local-inference testing, you have headroom the Jetson version doesn't — which means Nemotron 3 Nano 4B on a desktop card is less "pushing the limit" and more "using 20% of what you have," with room to run it alongside other workloads.
Running it locally
The GGUF release is the path of least resistance if your stack is llama.cpp-based, the way most of ours is:
llama-server -hf nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q4_K_M -c 0 --alias my_model -ngl 999 --port 5000 --host 0.0.0.0
That one-liner pulls the Q4_K_M quant straight from Hugging Face and serves it on an OpenAI-compatible local endpoint — -ngl 999 pushes every layer onto GPU, which a 4B model at Q4 has zero trouble affording on any card with 8GB or more. If you'd rather go through Ollama or LM Studio instead of raw llama.cpp, both are covered in our run-llama-locally guide, and the quantization tradeoffs behind that Q4_K_M tag are broken down in our quantization guide.
The license line worth reading twice
"Commercial use permitted" is doing a lot of work in NVIDIA's Nemotron Open Model License, and it's not the same thing as a true open-source license — no copyleft requirement, but also not the unrestricted MIT/Apache posture some rival small models ship under. If you're building something you plan to redistribute or fine-tune and re-release, read the actual license text rather than trusting a summary blog post — ours included.
Don't confuse it with its siblings
NVIDIA shipped the Nemotron 3 family as a set — Nano, Nano Omni, and larger Super and Ultra tiers — and the coverage online blurs them constantly. Nemotron 3 Nano Omni is a separate, multimodal variant built to see, hear and read in one model; it's a different architecture conversation than the text-focused 4B this guide covers. If you land on a benchmark or a video that's actually reviewing Omni or the 30B-A3B tier and cite it against the plain 4B, you'll draw the wrong conclusion about what to expect. Check the exact model name in whatever source you're reading — "Nemotron 3 Nano" alone isn't specific enough to know which one you're getting.
Where it sits against Laguna XS 2.1
Both models chase the same idea from different angles: get real capability out of a small active-compute footprint. Nemotron 3 Nano 4B does it with architecture — swap attention for Mamba. Laguna XS 2.1 does it with routing — a 33B MoE model that only activates 3B per token. If your job is coding-agent work specifically, Laguna XS 2.1's SWE-bench numbers are the more direct comparison point. If your job is general on-device inference — assistants, summarization, structured output — Nemotron's MATH500 and instruction-following scores are the more relevant read.
How we assessed this
Architecture and benchmark figures are pulled directly from NVIDIA's own Nemotron 3 Nano model card and Hugging Face documentation, not secondary aggregator summaries. We have not yet run the 4B model on our own RTX 4080 to independently verify the Jetson-reported throughput numbers translate to desktop hardware; NVIDIA's figures here are vendor-reported, and we'll fold in our own bench numbers once we've put real hours on it.
Frequently asked questions
▸What makes Nemotron 3 Nano 4B different from a normal 4B model?
The architecture. Instead of the usual all-transformer stack, its 42 layers split into 21 Mamba-2 layers, 17 MLP layers, and just 4 attention layers. Mamba layers scale near-linearly with sequence length instead of the quadratic cost of full attention, which is why NVIDIA can claim meaningfully faster inference at a given parameter count.
▸Can I run this on my RTX 4080?
Yes — 4B parameters at Q4_K_M GGUF fits comfortably in well under 8GB of VRAM, which an RTX 4080's 16GB clears without quantization gymnastics. It's built primarily for Jetson edge hardware, but the GGUF release runs on any CUDA card llama.cpp supports.
▸Is Nemotron 3 Nano 4B open source?
It's commercial-permissive under the NVIDIA Nemotron Open Model License, not a copyleft-style open-source license and not a no-strings MIT/Apache release either. Read the actual license text before a commercial deployment — 'commercial use permitted' and 'open source' aren't the same claim, and vendors sometimes blur the two.
▸Where did Nemotron 3 Nano 4B come from?
NVIDIA pruned and distilled it from the larger Nemotron Nano 9B v2 using their Nemotron Elastic framework — structured pruning down to 4B rather than training a new small model from scratch. That lineage is common in the current wave of small efficient models; the parent model's training quality carries over more than a from-scratch 4B typically achieves.
The 5 best AI video finds, every week
New models, tested prompts, and what actually worked in our production — one short email a week. No spam, unsubscribe anytime.
Written by Derek Holt
Local AI & Hardware Writer
Runs the site's local-inference rig and benchmarks every GPU, quant, and speed-stack claim on it personally before it goes in a guide. Will not shut up about VRAM bandwidth.
Explore these topics
Every guide, comparison and prompt library we have on each.
Keep learning
GPUs & Hardware · Local LLMs
GuidesNVIDIA RTX Spark for Local AI (2026): What It Actually Runs
128GB unified memory, a 20-core Grace CPU, a Blackwell GPU on one chip. What RTX Spark can actually run locally, what it costs, and who should wait to buy.
