NVIDIA serves 753B-parameter GLM-5.3 NVFP4 on two DGX Stations, peaking at 607.8 tokens/s

NVIDIA has published a walkthrough for serving GLM-5.3 NVFP4 across two DGX Stations with vLLM. Its AIPerf benchmark shows overall output throughput peaking at 607.8 tokens per second at concurrency 16. The post, dated Nov 13, 2026 and written by Fatima Fall, covers setup, a test request and a concurrency benchmark.

Why one GPU is not enough

The NVIDIA GLM-5.3 NVFP4 model is a quantized version of Z.ai's mixture-of-experts (MoE) model. It has 753 billion parameters, with 40 billion activated per token. NVIDIA notes that sparse activation reduces computation, but every weight must stay available because each token can be routed to different experts.

The checkpoint is about 464 GB. A single GB300 GPU has 252 GB of HBM3e, so the model does not fit. The tutorial keeps all weights in GPU memory across two Stations, with CPU weight offload disabled.

Deployment setup

• Interconnect: Two GB300 DGX Stations linked over ConnectX-8 RDMA. The Stations expose two direct rails, cx8r0 and cx8r1. The tutorial picks one for serving traffic, using cx8r1 as its example.
• Parallelism: Tensor parallelism (TP=2) splits the model's tensors across the two GPUs. Each GPU holds and computes roughly half the weights, and the two memories remain distinct.
• Roles: Station A (rank 0) hosts the API server. Station B (rank 1) runs only a worker. Both compute every request, and Station A returns the answer through a single OpenAI-compatible endpoint.
• vLLM features: The deployment enables multi-token prediction, FP8 KV-cache storage and prefix caching, which NVIDIA says maximize throughput under concurrent load.

Prerequisites

Both Stations must run Linux and pass the Connect Two DGX Stations playbook, including its GPU-memory RDMA test on both rails. Each needs SSH and sudo access with the same username, and at least 464 GB of free disk space for its own copy of the checkpoint, plus room for the vLLM container image and temporary files. Docker and the NVIDIA Container Toolkit come preinstalled with the DGX Station software stack.

Users must accept the NVIDIA Open Model Agreement and the original GLM-5.3 license terms listed on the model card. The benchmark uses SPEED-Bench, governed by the NVIDIA Evaluation Dataset License Agreement. Its prompts are downloaded from the gated cais/hle dataset on Hugging Face, so a Hugging Face account with a read token and accepted access terms is required.

Benchmark results

NVIDIA ran AIPerf on the SPEED-Bench Coding dataset. Overall output throughput peaked at 607.8 tokens per second at concurrency 16. At concurrency 32 it fell to 385.0 tokens per second, with a mean request latency of 453.5 seconds.

These figures come from NVIDIA's own tests on its own hardware and are presented as a baseline for this configuration.

Next steps from NVIDIA

NVIDIA points readers to the Connect Two DGX Stations playbook, its Local AI recipe repository for current recommended configurations, and the vLLM serving documentation. It also suggests rerunning AIPerf with custom prompts to find the best concurrency for a given workload.

What this means

For teams running models of this size locally, the key constraint in this setup is memory: the 464 GB checkpoint cannot fit in one GB300's 252 GB of HBM3e, so TP=2 across two Stations is how NVIDIA keeps every weight in GPU memory. The benchmark shows throughput does not rise steadily with load. In NVIDIA's test, 16 concurrent requests gave the highest output throughput, and 32 gave lower throughput alongside a mean request latency of 453.5 seconds. NVIDIA's own advice to rerun AIPerf with custom prompts reflects that the best concurrency depends on the workload, so the SPEED-Bench Coding numbers are a starting point rather than a general rule.