Speculative decoding on GB10: a wider draft speeds up one stream and slows sixteen

The measurements this piece rests on are not mine. In August 2026 darkdatter swept SGLang’s --speculative-num-draft-tokens for Qwen3.8-27B on an ASUS DGX Spark and published the recipe and results in gb10-repo; kosztyua reproduced the key numbers on a second GB10 in pull request #1. My part is smaller: the recipe ran on my own Spark on 30 August, and in September I tried a faster operating point of my own for about a week. Their data shows the trade-off cleanly; mine points the same way and proves less.

One flag moves the server in two directions at once. In darkdatter’s sweep, at the default of 8 a single stream decoded at 61.3 tokens per second and sixteen concurrent streams at 429.2 in aggregate. At 16 the single stream rose to 78.6 (+28%) and the sixteen streams fell to 384.8 (−10%). Which width is right depends on who is waiting.

Why checking is nearly free

In speculative decoding a small draft model proposes a block of tokens and the large target model checks the whole block in one forward pass, keeping drafted tokens up to the first one it disagrees with. The previous essay argued that decode on this machine is bandwidth-bound: the expensive part of a forward pass is reading the weights, so checking sixteen drafted tokens costs about what generating one does.

Two numbers describe how well it works. accept_len is the mean number of tokens accepted per verification step, and it sets the speed-up. accept_rate is the fraction of drafted tokens that survive, and it says how much draft work was thrown away. A wider draft raises the first only while the drafter keeps guessing right; it lowers the second almost every time.

The sweep

darkdatter’s setup was one ASUS DGX Spark (GB10, 128 GB unified memory) with nothing else running, driver 580.173.02 and CUDA 13.0. The target model is RadixArk’s NVFP4 quantisation of Qwen3.8-27B, the draft model z-lab’s DFlash2, the engine SGLang from LMSYS. The recipe pins LMSYS’s official DFlash2 image and uses Mia AI Lab’s Qwen3.8-27B toolkit for its launch script, SGLang overlay and NVFP4 head patch. Throughput is code generation at temperature 0, counted as completion_tokens over wall time. The server flags:

--kv-cache-dtype fp8_e4m3 --mamba-ssm-dtype bfloat16 --mem-fraction-static 0.85
--context-length 262144 --max-total-tokens 1048576
--max-running-requests 16 --max-mamba-cache-size 80 --cuda-graph-max-bs-decode 16

All widths ran on one build; the draft-16 single-stream figure is from a fresh boot:

Single-stream speed climbs to a plateau between 12 and 16, then falls. Aggregate throughput at sixteen streams peaks at 10 and falls at every step after. accept_len rises to 9.52 at 16 and then drops to 8.48 and 8.40, while accept_rate falls the whole way, from 0.85 to 0.32. darkdatter’s reading is that past 16 the drafter cannot sustain longer correct runs, so the extra draft compute buys rejected tokens.

That leaves two optima: 16 for one interactive user, 10 for a server under concurrent load (435.1 against 384.8 in aggregate). One server cannot have both.

Why the optima diverge

With one stream, decode leaves most of the GPU’s compute idle while it waits on memory, so drafting and verifying more tokens spends compute nobody else wanted. With sixteen streams in a batch, the batch is already using that compute, and darkdatter’s explanation is that wasted draft work then competes with real work from other requests.

Divided by sixteen, the sweep’s aggregates give each of sixteen users 26.8 tokens per second at draft 8 and 24.1 at draft 16. The lone user gets 61.3 and 78.6.

How sure anyone can be

Less sure of the magnitude than the table suggests. Draft 16 has three single-stream measurements: 85.7 tokens per second inside darkdatter’s sweep, 78.6 on a fresh boot of the same machine, and 84.4 on kosztyua’s. The table carries 78.6, which darkdatter’s own notes call probably the low end of the spread rather than its centre. darkdatter measured boot-to-boot variance in single-stream throughput at about 8%, against under 2% between runs within one boot, so 12, 14 and 16 cannot be told apart. Only the broad shape is trustworthy.

kosztyua’s reproduction on 23 August used the same benchmark script and model pins, the official LMSYS DFlash2 image, and a within-boot A/B with only this flag changed. Going from draft 8 to 16 took single-stream throughput from 59.6 to 84.4 tokens per second (+41.6%) and the 16-stream aggregate from 464.6 to 393.6 (−15%). accept_len landed within 3% of darkdatter’s at both widths (6.82 and 9.25), so the mechanism matches, though the single-stream gain came out larger than +28%.

Quality should not move, because the target verifies every draft token and keeps only what it would have produced itself. kosztyua also ran a quality check at both widths, HumanEval pass@1 with 164 problems run against their unit tests:

Two problems up in one mode, two down in the other. That is inside the noise darkdatter documents: even at temperature 0 the server flips two or three HumanEval problems between identical runs, because dynamic batching and speculative decoding change the order of floating-point reductions.

What ran on my own Spark

On 30 August the recipe ran on my machine: darkdatter’s unmodified benchmark script at commit 2db1ecd, with a profile for four agents (four requests, 20 state slots, draft 10, FP8 KV cache, memory fraction 0.85). One stream decoded code at 71.7 tokens per second; darkdatter’s sweep has 65.2 at draft 10, on a server configured for sixteen requests. Four streams reached 210.6 in aggregate, 52.7 each.

On 2 September my coding agent set up a “fast lane” for one or two users at a time. Against that profile it changed four settings at once: draft 10 to 16, FP8 to BF16 KV cache, request cap 4 to 3 (state slots 20 to 15), and token pool 1,048,576 to 524,288. Each level below is the median of three groups of 512-token code requests at temperature 0:

This A/B used its own benchmark, 512-token code requests with the median of three groups per level, so its 55.69 single-request figure is not comparable with the 71.7 from 30 August.

Correctness gates passed: 20 of 20 direct checks (arithmetic and a structured tool call) and six of six agent file tasks across the three levels. The recorded verdict was a trial for one and two users, with three as a hard ceiling. It ran as a trial for about a week, and then I moved to another model. The launch script, raw JSON and report are in my recipes repository under profiles/qwen3.8-27b-dflash2/.

Why the KV type changed in the same step is not in my notes. The pool change is arithmetic: BF16 takes twice the bytes per token of FP8, so halving the pool kept the KV cache’s byte budget where it was. The direction matches the sweep, faster for few users and slower at the cap. But with four changes at once I cannot say how much of any row is the draft width, and the one-user gain of 7.9% sits inside the 8% boot-to-boot band darkdatter measured.

Counting the right thing

Two measurement traps in this material would each change a headline:

• Count completion_tokens, not stream events. DFlash2 sends about 3.75 tokens per server-sent event, so a client that counts events as tokens reads about 3.75 times too low. One README I worked from says event counting inflates throughput by about four times; the error runs the other way. Mia AI Lab’s toolkit notes have it right: they measured the stream at about eight events a second whatever the throughput, and say event counting “under-reads tok/s ~4×”.
• Never quote two widths as one configuration. 78.6 single-stream (draft 16) and 480.7 aggregate (draft 8, from a separate concurrency run) read like one server’s spec sheet, and no server delivers both. kosztyua’s review flagged darkdatter’s methodology header for mixing them; the results file now labels which rows were measured at which width.

One warning stays open. At draft 16 SGLang logs DFLASH block size mismatch, because the public draft config declares a block of 8 while the runtime verifies 16. darkdatter’s sweep ran in this mode and my fast lane’s gates passed, but the quality evidence for draft 16 is kosztyua’s one HumanEval run per mode and those gates. None of it covers a long agent session.

An AI coding agent ran the 30 August rerun and set up and ran the fast-lane trial on my machine; the sweep and the reproduction are darkdatter’s and kosztyua’s, credited above.