How to deploy NVIDIA Nemotron-3-Nano-Omni-30B-A3B-Reasoning NVFP4 on Lambda

Performance

NVIDIA Nemotron-3-Nano-Omni-30B-A3B-Reasoning NVFP4
1× NVIDIA B200 GPU or 1× NVIDIA H100 GPU · vLLM TP1
12,479 output tok/s maximum throughput on B200
762 tok/s/user at interactive latency on B200

Benchmark summary

Background

NVIDIA Nemotron-3-Nano-Omni-30B-A3B-Reasoning is a natively multimodal member of the Nemotron-3-Nano family, combining a Mamba2-Transformer hybrid Mixture-of-Experts (MoE) language backbone with two purpose-built encoders: CRADIO v4-H for vision and Parakeet (parakeet-tdt-0.6b-v2) for audio. It accepts video, audio, image, and text input and produces text output only. The "Reasoning" variant is post-trained for extended chain-of-thought.

Per NVIDIA's model card, the backbone totals approximately 31 billion parameters with roughly 3 billion active per token, and the model supports a context window of up to 256,000 tokens. NVIDIA publishes three quantization variants — BF16 (62 GB), FP8 (33 GB), and NVFP4 (~21 GB) — and reports that the quantized variants stay within roughly one point of the BF16 baseline across tested benchmarks. This card covers the official NVFP4 checkpoint.

Model specifications

Overview

• Name: NVIDIA Nemotron-3-Nano-Omni-30B-A3B-Reasoning (NVFP4)
• Author: NVIDIA
• Architecture: Mamba2-Transformer hybrid Mixture-of-Experts, with a CRADIO v4-H vision encoder and a Parakeet audio encoder
• License: NVIDIA Open Model License Agreement

Specifications

• Total parameters: 31B total (3B active per token)
• Context window: up to 256,000 tokens
• Modality: Natively multimodal — video, audio, image, and text input; text-only output
• Precision: Native NVFP4 — official nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 checkpoint, ~21 GB on disk

Hardware requirements

• Minimal deployment:
  • 1× NVIDIA Blackwell GPU (e.g. 1× NVIDIA B200 GPU) for the NVFP4 checkpoint. The same checkpoint also runs on 1× NVIDIA H100 GPU via a non-native FP4 execution path (Hopper has no native FP4 tensor cores).

Deployment and benchmarking

Deploying NVIDIA Nemotron-3-Nano-Omni-30B-A3B-Reasoning (NVFP4)

NVIDIA Nemotron-3-Nano-Omni-30B-A3B-Reasoning (NVFP4) was benchmarked on a single NVIDIA Blackwell or NVIDIA Hopper GPU with vLLM, tensor-parallel size 1, text-only chat completions (--language-model-only-style path; the vision/audio encoders were not exercised).

1. Launch an instance with 1× NVIDIA B200 GPU (or 1× NVIDIA H100 GPU) from the Lambda Cloud Console using the GPU Base 24.04 image.
2. Connect to your instance via SSH or the JupyterLab terminal. See Connecting to an instance for detailed instructions.
3. Start the inference server:

docker run --gpus all -p 8000:8000 --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.20.0 \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \
  --served-model-name nvidia/nemotron-3-nano-omni \
  --tensor-parallel-size 1 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code

Verify the server

curl -X GET http://localhost:8000/v1/models \
  -H "Content-Type: application/json"

You should see nvidia/nemotron-3-nano-omni listed in the response.

Benchmarking

Results are in the benchmark summary table above. Both GPU lanes were driven with AIPerf (NVIDIA Dynamo AIPerf 0.12.0) at approximately 8,192 input / 1,024 output tokens, 512 requests per concurrency point, swept across nine concurrency levels (1–256).

Next steps

Upstream

• Download NVIDIA Nemotron-3-Nano-Omni-30B-A3B-Reasoning NVFP4 on Hugging Face