320 Billion Parameters, Two Desktops, One User at a Time

GLM-5.3-Flash is a 320-billion-parameter Mixture-of-Experts model, natively multimodal, with 18 billion parameters active per token. LibertAI quantized it to NVFP4: 598.5 GiB of BF16 weights down to 181 GiB, a 70% reduction, at a round-trip cosine of 0.99665.

That is the difference between "needs a GB200 rack" and "runs on two boxes under a desk". I got it serving on my two NVIDIA DGX Sparks, and it works: correct reasoning traces, working tool calls, coherent output.

The headline is the size. The useful findings are underneath it, and several of them are more restrictive than the spec sheet suggests.

LibertAI deliberately published no throughput numbers. As far as I can tell, the figures below are the first measured for this checkpoint on any hardware.

The setup

Model: LibertAIDAI/GLM-5.3-Flash-NVFP4, a quantization of zai-org/GLM-5.3-Flash

Weight-only NVFP4: only the routed-expert FFN tensors are quantized, which is 97% of the parameters. Everything outlier-sensitive stays BF16 — both attention flavours, the vision tower, shared experts, routers, embeddings, the MTP head

Hardware: two DGX Sparks (GB10, sm_121), 121 GiB usable memory each, connected over RoCE

Engine: SGLang, tensor parallel across both machines

The two machines are called Jean-Luc and Kathryn.

It fits, at 90.5 GiB of weights per machine

181 GiB split across TP=2 is roughly 90.5 GiB of weights per Spark. At a static memory fraction of 0.84, about 17 GiB remains free per node after weights, KV cache and state cache. Cold load is 465 to 480 seconds — considerably faster than I expected for a model this size.

14.2 to 14.6 tokens per second, and remarkably stable

Measured across three prompts at 192 tokens each, median:

Run	Values	Median
First load	14.4 / 14.6 / 14.6	14.6 tok/s
Second load	14.0 / 14.5 / 14.2	14.2 tok/s

The spread is unusually tight. For a 320B model on desktop hardware this is workable. It is not a speed at which you stop noticing you are waiting.

Concurrency buys exactly nothing

This is the finding that changes how you use it.

Eight requests in flight produced 14.4 tok/s aggregate — the same as a single request. Not degraded, not improved. Identical.

The reason is architectural. GLM-5.3-Flash has 34 KDA linear-attention layers, and each one needs a per-request recurrent state. That state cache, not the KV cache, is the binding constraint. With max-running-requests at 2 and a nine-slot state cache, everything serialises.

Context length and concurrency compete directly for the same memory. If you hit the error mamba cache too small ... max_num_reqs=0, lower the concurrency rather than the context — the state cost is per request, so concurrency is the cheaper thing to give up.

In practice: this is a single-user model. Not a shared team endpoint.

Prefill is the real bottleneck, not generation

4,020 prompt tokens took 41 seconds. That is 98 tok/s of prefill.

Extrapolated to the configured 65,536-token context, that is roughly eleven minutes before the first output token appears. The model card advertises a 1 million token context. On this hardware that number is a theoretical ceiling, not a usable one.

If you plan long-document work here, plan around prefill, not around the 14 tok/s of decode.

Speculative decoding does not work with this checkpoint

MTP would have been the obvious lever. The checkpoint keeps the MTP head in BF16, so it should work. It does not, and the failure is reproducible:

glm5_next_nextn.py:84   load_weights
  glm5_next.py:1705     load_weights
    fused_moe_triton/layer.py:703  _load_w13
RuntimeError: The size of tensor a (4096) must match the size of tensor b (2048)
              at non-singleton dimension 1

4096 is exactly twice 2048. That is the signature of a BF16 tensor being loaded into an NVFP4-sized slot, because NVFP4 packs two values per byte. The checkpoint deliberately keeps the MTP head in BF16; SGLang's NextN module sizes its MoE parameters as NVFP4.

The error occurs at weight loading, not at runtime. That is why draft depths of 1, 2 and 3 all fail identically — not a single token is ever speculated. Reproduced on both the stock image and my patched one.

Worth being precise about the attribution: the model card never claims otherwise. Its line about speculative decoding sits in the vLLM section, and its verified SGLang recipe carries no speculative flags. The wrong assumption was mine.

GB10 needs a kernel patch, or nothing runs at all

The stock image loads the weights, runs sparse-attention prefill, and then dies at decode warmup:

tvm.error.InternalError: Failed to set the allowed dynamic
shared memory size to 169984

GLM-5.3-Flash is NoPE — its rope head dimension is zero, so the tail dimension is zero. That routes the sparse forward pass to a kernel whose default tile (block 64, two stages, 256 threads) requests 169,984 bytes of dynamic shared memory. GB10 permits 101,376 bytes. Datacenter Blackwell has the headroom; workstation Blackwell does not.

This is not fixable from the command line — the tile is baked into the kernel factory's defaults. The fix is a one-line override at the call site: LibertAI's verified GB10 tile of block 32, one stage, 128 threads, guarded on sm_12x so datacenter cards keep the faster default.

Two details turn that from "it worked" into "it is correct". The three numbers are coupled: threads must drop to 128 before the block can drop to 32, or the fragment layouts become unsatisfiable. And the HIP branch twenty lines above already uses exactly 32 and 128 for non-gfx95 cards.

The trap: a small token budget returns an empty answer

The model thinks by default, and the reasoning trace is paid out of the same token budget as the answer. Measured, identical prompt:

max_tokens	finish_reason	Result
300	length	all 300 tokens reasoning, content empty
900	stop	609 tokens, correct answer, 2,491 characters of reasoning

For a one-sentence question. Budget tightly and you get an empty response and conclude the model is broken. Budget around 1,000 tokens, or disable reasoning per request.

What works cleanly

The glm45 reasoning parser splits correctly: the answer lands in content, the derivation in the reasoning field

Tool calling with glm47: a well-formed call, finish reason tool_calls

One trap worth naming: SGLang's own auto-detection suggests glm45 as the *tool* parser. That parser fails silently on GLM-5.x — no error, empty content, null tool calls. Set glm47 explicitly

Conclusion

A 320B model runs on two desktop machines, with correct reasoning and working tool calls. That is genuinely remarkable.

The honest usage envelope is narrower than the specifications imply: one user, short prompts, 14 tokens per second, no speculative decoding. Not the model for a shared endpoint, and not the model for long documents.

The two available tuning levers turned out to be MTP, which is broken, and CUDA graphs, which were already at their best setting. The shipped configuration was already the optimal one. That is an unspectacular result, but a clean one — and I would rather publish that than a tuning story I did not actually measure.