What 128 GB of unified memory buys you: serving LLMs on a DGX Spark

In July an evaluation launcher my coding agent had written started vLLM 0.23 with --gpu-memory-utilization 0.85. vLLM sees 121.7 GiB of device memory on a DGX Spark, so for Qwen3-8B that flag set aside about 103 GiB before the first request arrived. The BF16 weights took 15.27 GiB. vLLM turned 86.89 GiB into KV cache, 632,720 tokens, and its log put that at room for 19.31 sequences of 32,768 tokens. The workload, budgeted at four sequences of that length with 25% headroom, needed 22.5 GiB. About 64 GiB of KV cache had nothing to hold, on a machine where that memory is also the operating system’s RAM.

One pool for everything

The GB10 has no separate VRAM. CPU, GPU and operating system share 128 GB of unified memory, so free -g shows what the GPU has allocated, and a “GPU memory fraction” is a fraction of the whole machine.

High fractions starve the operating system. darkdatter’s published recipe for Qwen3.8-27B on this hardware (gb10-repo) recommends 0.85 on a first boot, because both that project and the toolkit it used recorded machines “unrecoverably wedged during weight load” at higher fractions.

Memory also outlives the process you kill. When vLLM’s parent process dies, its EngineCore workers stay alive and keep their memory, and pgrep -f "VLLM::" does not find them: the thread carries that name, the command line does not. Kill them by PID.

Even then the memory comes back slowly, and vLLM checks free memory before it starts. A server launched too soon after the last one stops at initialisation with Free memory on device ... is less than desired GPU memory utilization. The launcher was changed to wait until enough memory is free.

Capacity is not speed

NVIDIA specifies the GB10’s memory bandwidth at 273 GB/s. Generating one token for one stream means reading every active weight once, so single-stream decode is capped at roughly bandwidth divided by the bytes of active weights. For Qwen3-8B in BF16 that is 273 GB/s over 16.4 GB (the 15.27 GiB above), about 17 tokens per second at best. That is arithmetic, not a measurement, and spare capacity does nothing for it.

Two things get around the cap. A mixture-of-experts model reads only the experts it routes to: Qwen3-30B-A3B touches about 3B parameters per token, so it decodes like a small model while all 30B stay resident. Memory pays for the total, bandwidth for the active part.

The other is batching. One pass over the weights serves every sequence in the batch. On 30 August Qwen3.8-27B ran on my machine under SGLang, speculative decoding on, through darkdatter’s public benchmark script: one stream decoded code at 71.7 tokens per second, four streams at 210.6 in total, 52.7 each. The machine produced nearly three times as much; each user got about a quarter less.

Size the KV cache from the workload

Weights are the easy part: parameters times bytes per parameter, about 2 in BF16, 1 in FP8 and 0.5 in NVFP4, plus the scales that 4-bit formats carry. Quantisation shrinks the footprint and the bytes each token has to read at the same time.

The KV cache is the part that depends on the workload. Every token of every live sequence stores a key and a value vector per layer and KV head:

$$\text{KV bytes} = 2 \times \text{layers} \times \text{KV heads} \times \text{head dim} \times 2\,\text{B} \times \text{context} \times \text{sequences} \times \text{headroom}$$

The 2 B is one BF16 element; with an FP8 KV cache it becomes 1. The launcher used a headroom of 1.25. Layers, KV heads and head dimension come from the model’s public config.json; context and sequences come from the workload, here 32,768 tokens and four sequences.

The 8B figure is exactly 24,159,191,040 bytes, the 22.5 GiB from the opening. The model with almost four times the parameters needs a third less, because it has half the KV heads and only a third more layers. Parameter count says little about KV size.

vLLM 0.23 has a flag for this: --kv-cache-memory-bytes loads the weights and gives the KV cache exactly that many bytes, whatever the total. From 4 July the launcher computed the number with a small script that reads config.json and applies the formula, and passed both byte counts above to vLLM unchanged.

The fraction flag still matters, because of the startup check: vLLM compares free memory with the requested fraction even when it has a byte budget. That evening five launches in a row stopped at initialisation, each finding about 81.9 of 121.69 GiB free against the 109.52 GiB that 0.9 asked for. The next launches set 0.5 and started.

Where the formula stops working

The formula assumes every layer is attention. Qwen3.8-27B is a hybrid: alongside full-attention layers it has Gated DeltaNet linear-attention layers, which keep a fixed-size state per request instead of a cache that grows with context. SGLang holds that state in a separate pool, sized in slots with --max-mamba-cache-size, and the pool caps concurrency.

The measurements in this section are darkdatter’s, from an ASUS DGX Spark in August 2026, published in gb10-repo. With the DFlash2 draft model for speculative decoding attached, each request needs five slots rather than four, because the draft’s verification step takes one extra. SGLang clamps running requests to the pool size divided by five, so sixteen concurrent requests need 80 slots:

At 80 slots the pool is 5.70 GiB of SSM state, 9.56 GiB of intermediate state and 0.38 GiB of convolution caches,¹1 SGLang’s logs label these sizes GB, but its memory-pool code divides by 1024³, so they are GiB. and those sizes hold only with --mamba-ssm-dtype bfloat16. Left unset, the state resolves to float32 and the pool doubles to 30.9 GiB. The slots are also why darkdatter’s concurrency run peaked at sixteen streams: sixteen was the cap, and at 32 streams requests queued (469.8 tok/s). My own two profiles for this model follow the five-slot rule, 20 slots for four requests in August and 15 for three in September.

darkdatter also found that the memory fraction made no measurable difference. SGLang’s --mem-fraction-static at 0.82, 0.85 and 0.90, with the KV pool capped at 1,048,576 tokens, gave 60.0, 61.8 and 62.3 tokens per second for one stream and 480.7, 472.0 and 468.7 for sixteen, all within run-to-run noise. Uncapping the pool at 0.90 grew it to 1,496,047 tokens; peak use never passed 67% of that, and the lost headroom cost 18% at 32 streams. The conclusion in darkdatter’s notes: on this model concurrency is bound by state slots and long context by prefill, never by memory capacity.

Qwen3.6-35B-A3B, the model I served daily in July, has the same kind of linear-attention layers. Serving it on vLLM under agent load produced the hang I describe in a separate note.

What the capacity does buy

Context first. Qwen3.8-27B serves a 262,144-token window, and prefill is the price. On my machine on 30 August, with darkdatter’s unmodified benchmark script:

The rate falls as the prompt grows, and a 120K-token prompt waits a minute and a half for its first token. darkdatter measured 94.89 s for the same prompt, and kosztyua 88.68 s on a second GB10.

Concurrency does not rescue long prompts. In darkdatter’s test, four, eight and twelve streams of 83K unique tokens each finished in 277, 556 and 833 seconds: 69.4 seconds per stream, dead linear, because prefill took one new sequence at a time. At twelve streams the pool held 998,767 unique KV tokens. A sixteen-stream run failed, but not on memory: the model was too busy prefilling to answer a 5-second health probe, and the supervisor restarted it. The first version of that test had built every prompt from the same filler, so SGLang’s prefix cache deduplicated them and the timings were cache hits; unique content per stream and an assertion that cached_tokens == 0 fixed it.

That cache is also where capacity pays off directly. Prefix reuse keeps the KV of an earlier prompt so a repeat skips its prefill. On my server on 2 September, an exact repeat of a 68,317-token prompt cut time to first token from 44.07 s to 0.65 s, 68 times faster. A workload that resends a long prefix every turn, as coding agents do, gets that saving for as long as the prefix stays in the cache.

The rules in my machine notes

These sit in the notes my coding agent loads at the start of every session:

• Small models (8B to 12B) on vLLM: --gpu-memory-utilization 0.35 to 0.45, or better, --kv-cache-memory-bytes computed from config.json and the workload. Going lower costs concurrent conversations, not accuracy.
• 30B and up: 0.80 to 0.85, because the weights are most of the reservation anyway. The Qwen3.6-35B-A3B server I used daily in July ran at 0.80 with a 131,072-token context and four sequences.
• After killing a server: find the workers by PID and wait for the memory to come back before starting the next one.

My two SGLang profiles for the hybrid model add five state slots per request, --mamba-ssm-dtype bfloat16, a memory fraction of 0.85 and a token cap.

Nothing in that list makes a single stream faster; 273 GB/s sets that ceiling. Speculative decoding works around it, and its draft width trades one user’s latency against everyone else’s throughput. That is the follow-up.

An AI coding agent wrote the evaluation launcher and the KV-sizing script, ran the vLLM jobs quoted here and the 30 August Qwen3.8-27B rerun, and set up and ran the 2 September trial on my machine. The hybrid-state, memory-fraction and long-context concurrency figures are darkdatter’s, credited above.

Notes

1. SGLang’s logs label these sizes GB, but its memory-pool code divides by 1024³, so they are GiB. ↩