Local LLM Benchmark Results on DGX Spark & 96–128 GB Edge AI Workstations

October 2026 update. These are the earlier (April–September 2026) results with older images. The new same-hardware round with vLLM 0.31.0, 128K context and 4 concurrent users is in Benchmarks, with copy-paste recipes. Notably, Nemotron 3 Super 120B-A12B now runs on vLLM 0.31.0 (16.1 tok/s, 23.0 with MTP, ~95 GB), so the vLLM failure described below applies only to the older gemma4-0505-cu130 image.

For the quick start guide see Home. For the original Spanish log see Results in Spanish.

This document contains the full benchmark results and technical notes for running Gemma 4, Qwen 3.6 and NVIDIA Nemotron 3 locally on the NVIDIA DGX Spark and equivalent high-memory edge AI hardware.

Hardware / software base

Critical GB10 limitation: no native FP4 compute. NVFP4 checkpoints run through the Marlin backend (--moe-backend marlin) in vLLM or the PyTorch backend in TensorRT-LLM, decompressing FP4 → BF16 at runtime. This limits throughput compared to FP4-native GPUs such as the B200.

Benchmark methodology

Text benchmarks used a ~120-token Spanish prompt, max_tokens=512, temperature 0.7 and streaming enabled. We report decode tok/s (excluding cold-start TTFT) and hot TTFT. Memory numbers reflect peak unified memory usage observed during the run.

Full results table

Total = parámetros totales del modelo · Activos = los que se ejecutan por token (clave para bandwidth del decode).

Key takeaways

• Current default (2026-09-01): Qwen3.8-Flash-Next NVFP4 hybrid — best quality/speed balance for agentic workflows on the Spark. ~37 tok/s warm single-stream, 117 @c=8 aggregate, GSM8K 97.27%, AIME26 98.75%, MoE 176B (6B activos), multimodal texto+imagen+video, 262K contexto nativo (500K YaRN). Recipe upstream blazux/qwen3.8-Flash-DGX (7 parches GB10 sobre vLLM release/qwen38next). Cold start ~14 min. Disco: ~139 GB.
• Fallback lite: Qwen 3.8 27B NVFP4 + DSpark k=14 — single-stream 30 tok/s fresh, 70-76 warm, 253 tok/s @c=16 aggregate. Multimodal, 262K contexto. Mejor que Qwen 3.6 cuando se necesita concurrencia para múltiples auxiliares Hermes/Opencode en paralelo.
• Single-stream champion (long-context): Qwen 3.6 35B-A3B nvidia NVFP4 + vLLM nightly with --attention-backend flashinfer, --kv-cache-dtype fp8, --moe-backend marlin, VLLM_TEST_FORCE_FP8_MARLIN=1 and VLLM_MARLIN_USE_ATOMIC_ADD=1. The Marlin backend avoids the broken CUTLASS FP4 path on GB10/SM121 and lets vLLM use torch.compile/CUDAGraph without the bmm_fp8 crash we saw earlier. With --max-num-seqs 1 we run the model’s full 262,144-token context window. This keeps ~75–76 tok/s while giving agents ~237K tokens of input budget plus 25K of output headroom. Sacrifica concurrencia (1 secuencia) — incompatible con agentes que corren múltiples auxiliares.
• For maximum speed (~50 tok/s): use Gemma 4 26B-A4B community + patch.
• Qwen 3.6 35B-A3B RedHatAI (~42 tok/s) remains a stable fallback if the nvidia checkpoint or nightly image are unavailable.
• Gemma 4 31B dense is not viable for fast interactive use on GB10 (~7 tok/s).
• TensorRT-LLM works for Qwen 3.6 if you manually quantize the base BF16 model to NVFP4 MLP-only (~34 tok/s), but it is slower than vLLM with the nvidia checkpoint.
• Speculative decoding did not help on general prompts; it may only help for very repetitive text or with a compatible MTP drafter.
• Nemotron-3-Nano-Omni runs at ~40 tok/s on vLLM and processes images. Audio failed due to container decoding issues, not the model itself.
• Nemotron-3-Super-120B-A12B is only stable with TensorRT-LLM. vLLM fails with CUDA OOM or hangs the system. See detailed analysis below.

Why vLLM (with Marlin) and not TensorRT-LLM for Qwen 3.6

We tested two paths for Qwen 3.6 35B-A3B on the DGX Spark:

TensorRT-LLM worked, but it was almost 2× slower than vLLM. The reason vLLM had been failing earlier was not vLLM itself, but the default FP4/CUTLASS backend on GB10 (SM121). Switching to the Marlin backend for NVFP4 solved the crashes and memory saturation, letting us keep vLLM’s speed.

What is Marlin?

Marlin is a GPU kernel backend optimized for running quantized weights — especially 4-bit (INT4/FP4) — on NVIDIA GPUs. For MoE models like Qwen 3.6, vLLM exposes it via --moe-backend marlin.

The GB10 in the DGX Spark has no native FP4 compute (SM121 lacks a working FP4/CUTLASS path for these checkpoints). Marlin works around that by:

1. Reading the FP4/quantized weights from disk.
2. Decompressing them to BF16 at runtime inside optimized CUDA kernels.
3. Using its own GEMM kernels instead of the broken CUTLASS FP4 kernels.

On newer GPUs with native FP4 (e.g. B200), the CUTLASS FP4 backend would be faster. On the Spark, Marlin is the only path that gives both speed and stability for the NVIDIA NVFP4 checkpoint.

The extra environment variables force vLLM to select Marlin-compatible kernels for FP8/Marlin fused ops and avoid atomic-add issues on this architecture:

-e VLLM_TEST_FORCE_FP8_MARLIN=1 \
-e VLLM_MARLIN_USE_ATOMIC_ADD=1

Without Marlin, vLLM falls back to CUTLASS FP4 kernels that crash or hang on SM121 — exactly the bmm_fp8 / CUDNN_STATUS_EXECUTION_FAILED_CUDA_DRIVER errors we saw earlier.

Why Nemotron-3 Super 120B-A12B fails on vLLM

Fixed in vLLM 0.31.0: see Benchmarks and the nemotron3-super-120b-nvfp4-mtp-v031 recipe.

We attempted twice to serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 with vllm/vllm-openai:gemma4-0505-cu130.

First attempt

The container loaded all 17 checkpoint shards (~75 GB on disk) in ~8 minutes. The last logs showed:

Loading weights took 489.67 seconds
WARNING ... Your GPU does not have native support for FP4 computation ...
Using MoEPrepareAndFinalizeNoDPEPModular
Using uncalibrated q_scale 1.0 ...

Before the HTTP server came up, the Spark hung and had to be rebooted. Root cause: two llama-server processes running Qwen GGUF models were consuming ~76 GB of unified memory, leaving insufficient room for vLLM.

Second attempt (after freeing memory)

After stopping qwen27-local.service and qwen35-local.service, the system had ~117 GB free. However, vLLM failed immediately while initializing the EngineCore:

torch.AcceleratorError: CUDA error: out of memory
  File ".../vllm/utils/mem_utils.py", line 108, in measure
    self.free_memory, self.total_memory = current_platform.mem_get_info(device)

vLLM could not even query available memory without hitting an OOM.

Diagnosis

The model uses a hybrid Mamba-MoE architecture (NemotronHForCausalLM) with:

• 88 layers
• 512 routed experts
• 22 active experts per token
• 1 shared expert
• Hidden size 4096, head dim 128, GQA with 2 KV heads

Although the checkpoint is ~75 GB on disk, decompressing FP4 → BF16 via Marlin, allocating KV cache under --gpu-memory-utilization 0.90, and compiling kernels with the vLLM V1 engine (torch.compile/inductor) exceeds the 128 GB unified-memory budget.

Why TensorRT-LLM works

The same checkpoint serves stably at ~14.7 tok/s and ~110 GB with:

trtllm-serve /models/nemotron --backend pytorch --max_seq_len 8192 --max_batch_size 1 --kv_cache_dtype fp8

TRT-LLM reserves a fixed memory budget (model + KV cache for the given seq_len/batch + activations) instead of vLLM’s percentage-based dynamic PagedAttention allocation.

Conclusion: on DGX Spark and equivalent 128 GB unified-memory hardware, Nemotron-3 Super 120B-A12B should only be used with TensorRT-LLM.

Extreme context scaling: Qwen 3.6 35B-A3B

For agentic workflows that ingest very long contexts (codebases, conversation history, RAG dumps, multi-turn agent traces) we tested how far nvidia/Qwen3.6-35B-A3B-NVFP4 can scale on the DGX Spark. We used vLLM nightly with:

--model /models/qwen3.6 \
--trust-remote-code \
--tensor-parallel-size 1 \
--attention-backend flashinfer \
--moe-backend marlin \
--gpu-memory-utilization 0.92 \
--max-model-len 262144 \
--max-num-seqs 1 \
--max-num-batched-tokens 32768 \
--enable-chunked-prefill \
--async-scheduling \
--enable-prefix-caching \
--load-format fastsafetensors \
--kv-cache-dtype fp8 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder

Environment variables passed to the container:

-e VLLM_TEST_FORCE_FP8_MARLIN=1 \
-e VLLM_MARLIN_USE_ATOMIC_ADD=1

The 1-sequence/262K configuration is the recommended default because it uses the model’s full context window (262,144 tokens) with a single concurrent sequence. We reserve 237K tokens for input and 25K for output to avoid ContextWindowExceededError on long agent sessions. Earlier iterations ran 240K/1 seq to leave extra memory headroom; after measuring memory use we moved to the full 262K window while keeping --max-num-seqs 1.

Single-sequence context scaling

Memory behavior

• At rest after loading: ~119 GB used / ~121 GB total, ~2 GB available.
• During ~237K prefill: ~120 GB used, ~1 GB available, minimal swap with the current 262K/1 seq config.
• Stable: no OOM, no hang, no reboot required during these tests.

Practical guidance for agents

• Average agent turn: OpenClaw / Hermes-style agents typically use 8K–32K tokens of active context per session.
• Conservative production setting: 1 session × 64K–128K context runs with sub-second TTFT and leaves comfortable headroom for LiteLLM/ASR.
• Maximum practical context per session: ~237K tokens of input with 1 concurrent session at the 262K config (leaving 25K for output).
• Do not use 2+ sessions at 262K unless the machine is dedicated to a single model and you can tolerate hangs from memory spikes.

Launch scripts

Ready-to-run recipes are in scripts/:

Run a benchmark after the container reports healthy:

python3 benchmarks/bench_model.py gemma-4-26b-a4b 512

Important technical notes

1. ARM64: always verify the Docker image has an arm64 manifest.
2. Marlin is required for MoE NVFP4 on GB10: use --moe-backend marlin. Native FP4 backends (CUTLASS/FlashInfer) may fail or produce NaN on sm_121.
3. FP8 KV cache saves memory but relies on checkpoint scaling factors; vLLM warns about accuracy if they are missing.
4. Prefix caching speeds up requests with shared context or repeated long prompts.
5. Tool calling: Qwen 3.6 requires --enable-auto-tool-choice --tool-call-parser qwen3_coder (the qwen3_coder parser is more robust for multi-turn than the older qwen3_xml); without a parser, vLLM returns XML in content and leaves the native tool_calls array empty. We omit --reasoning-parser for the nvidia/Qwen3.6-35B-A3B-NVFP4 checkpoint because it does not emit <think></think> tags; with the parser active, reasoning leaks into the reasoning field and content appears empty in agents.
6. max-num-batched-tokens: for multimodal input (Gemma 4), must be >= max_tokens_per_mm_item (e.g. 2496 for Gemma 4, 4096 by default).
7. TensorRT Model Optimizer on GB10: quantizing Qwen 3.6 from BF16 requires converting the VLM checkpoint to text-only and using total GPU memory (not free memory) because accelerate does not understand the 128 GB unified pool.
8. TRT-LLM PyTorch backend reads hf_quant_config.json; use --backend pytorch and --kv_cache_dtype fp8 for HF NVFP4 checkpoints.
9. Background services: stop VRAM consumers such as llama-server before launching large models. Two Qwen GGUF servers used ~76 GB in our tests and caused OOM/hangs.

Final recommendation

For agentic workflows on DGX Spark and similar 96–128 GB edge AI workstations:

• Qwen3.8-Flash-Next NVFP4 hybrid + vLLM release/qwen38next + 7 parches GB10 → default actual (2026-09-01). ~37 tok/s warm single-stream, 117 @c=8 aggregate, MoE 176B (6B activos), GSM8K 97.27%, AIME26 98.75%, multimodal texto+imagen+video, 262K contexto nativo (500K YaRN). Recipe blazux/qwen3.8-Flash-DGX. Cold start ~14 min. Disco ~139 GB.
• Qwen 3.8 27B NVFP4 + DSpark k=14 → fallback lite. ~30 tok/s fresh, 70-76 warm cache, 253 tok/s @c=16 aggregate, multimodal, 262K contexto (1M YaRN). Excelente para agentes con muchos auxiliares en paralelo.
• Qwen 3.6 35B-A3B nvidia NVFP4 + vLLM nightly → single-stream speed champion con long-context 262K. ~76 tok/s con --attention-backend flashinfer, --kv-cache-dtype fp8, --moe-backend marlin, VLLM_TEST_FORCE_FP8_MARLIN=1 y VLLM_MARLIN_USE_ATOMIC_ADD=1. Tool calling con qwen3_coder, image/video, 262K contexto con 1 sequence — ~237K input + 25K output. Sacrifica concurrencia (1 sola secuencia) — incompatible con agentes que corren múltiples auxiliares.
• Gemma 4 26B-A4B community + patch → ~49.5 tok/s, tool calling, low VRAM.
• Qwen 3.6 35B-A3B RedHatAI → ~42.2 tok/s, stable fallback if the nvidia checkpoint or nightly image are unavailable.
• Qwen 3.6 35B-A3B MLP-only NVFP4 (custom TRT-LLM) → ~34.4 tok/s if you prefer the official NVIDIA stack.
• Nemotron-3-Nano-Omni → ~40.0 tok/s, text + image, best official multimodal option.
• Nemotron-3-Super-120B-A12B → ~14.7 tok/s with TRT-LLM only, for quality-first workloads.

Gemma 4 31B dense should be reserved only for tasks where the dense model quality justifies ~7 tok/s.

Choosing the right model: