Contents
Data
| Part | GiB |
|---|---|
| FP4 weights | 75.92 |
| Memory reserve | 10.00 |
| Four stream windows | 19.12 |
| Prompt cache | 6.70 |
The server answers loaded in 180.1s and I have to leave it alone for three minutes. Not three minutes of nothing: during the load MemAvailable falls to about 36 GiB, and whatever else was using the GPU has to be gone before the unit starts. I host Qwen3.8-Flash-Next, NVIDIA’s quantisation-aware-distilled FP4 checkpoint (nvidia/Qwen3.8-Flash-Next-FP4-Local, revision b9ce18a6), through TensorFold v1.1.4’s Zig engine on one DGX Spark: GB10, 128 GB Unified memory shared with the host, free -g reporting 121 GB of it to Linux. The recipe is Mia’s AI Lab’s public single-Spark recipe, patches 0001-0017 on the stock engine.
The boot line from docker logs qwen38-flash-next-tf says exactly what the machine is spending:
[tensorfold] CUDA sm_121 device 0 (NVIDIA GB10, memory shared with the host):
model 75.92 GiB; 4 streams at once, 4.78 GiB each at a 262144-token window
(fp8 caches), of 25.9 GiB left after a 10.0 GiB reserve; prompts in passes of
up to 3072 rows, a 6.7 GiB prompt cache; own glue kernelsThat one line is the whole budget, and everything below is arithmetic on it.
The budget
The Quantisation-aware-distilled FP4 weights take 75.92 GiB. The container runs with TENSORFOLD_MEMORY_RESERVE_GIB=10, so the engine keeps a 10.00 GiB (10 GiB) reserve of MemAvailable for the operating system. What remains after weights and reserve, 25.9 GiB, is the KV cache pool. --parallel 4 with --context 262144 and --kv-cache fp8 gives each stream a full 262,144-token window, 4.78 GiB each: four windows come to 19.12 GiB, and 25.9 minus 19.12 leaves the 6.7 GiB (6.70 to two decimals) that the line calls the prompt cache.
Two details sit behind those numbers. The checkpoint on disk is about 96 GiB and loading takes about 81 GiB of memory, because the 26.8 GiB n-gram table is read from disk through the page cache instead of being held on the GPU: the recipe measured MemAvailable falling from 117.3 to 36.3 GiB at load with the vision helper on. My boot lands in the same place, which is why nothing else can be loading when the unit starts.
And the fp8 cache is what makes the arithmetic close at all: 17,888 bytes a token against 30,784 for bf16, per the recipe’s scripts/config.sh. The engine’s own table prices one fp8 window at 4.37 GiB; the boot line charges 4.78, and I did not isolate what the extra 0.41 GiB a stream covers. fp8 is also not free in quality: the recipe’s check puts it at about 98.8% top-1 agreement with a bf16 cache, so a free-running reply can differ from the exact one. I did not A/B the two caches on my own workload.
The prompt cache is a cap, not a reservation, as the recipe’s README puts it: it takes at most what the weights and the streams leave. Passing --prompt-cache-gib 12 changes nothing here. The 6.7 GiB exists because I run four streams and not five: one fewer window is one more window’s 4.78 GiB of prompt cache, which is why this build sits at 4.
Why four and not six
The engine admits each stream against the pool at start. The recipe measured where the wall is, and it is not far from where I sit: PARALLEL=8 at 262,144 tokens needs 38.27 GiB of cache and the engine answered “the CUDA memory budget leaves 27.22 GiB”. Six streams at 28.7 GiB fit only with a 7 GiB reserve, and the recipe’s run of that configuration kept its speed and exactness checks but left 0.2 GiB of prompt cache, where tools/prompt_reuse.py 30000 failed: a 32k-token follow-up resumed 4.7% of its prompt and prefilled again in 13.8 s. A server that re-reads every multi-turn history on every turn is not serving agents; it is re-reading them. Cutting the reserve is not free either: each GiB the reserve gives up is a GiB of page cache the n-gram table loses.
Data
| Streams | GiB for the windows |
|---|---|
| 4 | 19.12 |
| 6 | 28.7 |
| 8 | 38.27 |
Source: the recipe's own measurements and my arithmetic on the boot line
So the concurrency cap is not a policy I chose. It is what is left after the weights. The recipe’s own default is PARALLEL=5 at a 9 GiB reserve, and I run 4 at 10: one fewer stream bought a bigger reserve and, through issue #81’s arithmetic, one window’s worth of prompt cache.
The other thing that has to stop
The requirement in the recipe is “a DGX Spark with nothing else large on the GPU”, and the default window wants MemAvailable of about 107 GiB at start. In October, when I moved this endpoint from an earlier engine, the old server was stopped and its supervisor and timers disabled before TensorFold took the loopback port, so nothing could restart beside it. There is no co-tenancy on this box: another GPU model running during my load takes the memory the reserve is supposed to protect, and a server started on a shrunk pool carries that shrink for its whole life, since the pool is sized once, from MemAvailable after the weights land.
What the 6.7 GiB buys, and when it leaks
curl -s localhost:8888/metrics on the running server reports 247,711,917 prompt tokens asked and 192,893,509 restored from kept prompt states, 77.9% of them (my division, read at about 19:55 on 14 November; the counters move with every request). The done lines in the log carry prompt= and cached= per request, so I can split that by prompt length over 13,638 requests from just under three days of agent traffic.
Short requests, 2k-20k tokens: 8,676 of them, and the mean share of the prompt resumed from cache is 0.438, median 0.558. A short turn is served from cache about half the time, and when it is not, it prefills from scratch, which costs little.
Long conversations, 160k tokens and up: 123 requests, mean share 0.738, median 0.997, but the distribution is two lumps, not a hill. 89 resumed above 95% of their prompt. 8 resumed nothing at all: prompts of 160,219 to 166,784 tokens that found nothing in the 6.7 GiB and prefilled all of it. A conversation that outgrows the cache does not merely get less help; it loses the whole prefix, every turn, while a short one keeps getting served.
Data
| Share of the prompt resumed from cache | Requests | Mean TTFT (s) | Median TTFT (s) |
|---|---|---|---|
| Resumed above 95% | 89 | 5.6 | 0.99 |
| Resumed nothing | 8 | 185.1 | 174.1 |
The cost is in time to first token. A 160k+ request that hit the cache waited 5.6 s on average over 89 requests, median 0.99 s; a handful of slow warm requests pull the mean up, and I did not isolate why. One that fell out waited 185.1 s, median 174.1 s, over 8 requests, every one of them between 137.4 s and 274.5 s. A cold long turn is worth half a boot. The cumulative counter agrees with the leak: the prompt cache: lines show hits 12,941 against evicted 14,701, states evicted more often than served.
What I have not isolated is what evicts them. The log does not say which workload drove the 160k+ prompts, and the 8 cold requests could not be traced to a specific client from the record I read. The mechanism I can point at is arithmetic: at the fp8 price of 17,888 bytes a token, a 181,271-token prompt, which is also in the log, is about 3.02 GiB of the 6.7 GiB kept, so two such prefixes at once nearly fill it, and the least-used state goes. Four streams each holding a long window and a prompt cache that fits two long conversations is the same pool cut twice.
Two knobs move it and both cost the thing that makes the box useful. PARALLEL=3 leaves a second window’s 4.78 GiB to the cache, as the recipe’s issue #81 notes for its own build. A shorter --context leaves more still: the recipe calculates eight streams at 163,840 tokens at about 24 GiB of cache, more streams than I can run now, though it says the calculation was never run.
The open question is whether long agent conversations deserve their own slot count, or whether the answer is to keep histories shorter than the cache instead of buying capacity for them. I have run the server for just under three days with one configuration. The 185 s waits are real and the fix is untested.
Numbers from one machine and one server process running since 23:22 on 11 November. The PARALLEL=8 and PARALLEL=6 measurements are Mia’s AI Lab’s, on their Spark.