---
title: Fitting a 76 GiB FP4 model and four 262k windows into one GB10
kind: essay
date: 2026-11-14
author: Gürkan Özkan
url: https://gurkanozkan.ai/writing/hosting-a-large-model-on-gb10/
markdown: https://gurkanozkan.ai/writing/hosting-a-large-model-on-gb10.md
description: On a 121 GiB DGX Spark, the 75.92 GiB weights and four full 262,144-token windows leave a 6.7 GiB prompt cache, which is why concurrency stops at 4 and why long agent conversations stop hitting that cache while short ones keep hitting it.
topics:
  - ai
tags:
  - dgx-spark
  - gb10
  - serving
  - tensorfold
  - qwen
  - kv-cache
  - agents
concepts:
  - cold-start
  - inference-serving
  - kv-cache
  - llm-agents
  - prefix-caching
  - quantization
  - unified-memory
---

# Fitting a 76 GiB FP4 model and four 262k windows into one GB10

**Chart:** Cover: The weights and four full-length windows leave a prompt cache of 6.7 GiB. Source: the server's own boot line (docker logs) and my arithmetic on it.

*Where the memory goes at steady state, in GiB*

| Part | GiB |
| --- | ---: |
| FP4 weights | 75.92 |
| Memory reserve | 10.00 |
| Four stream windows | 19.12 |
| Prompt cache | 6.70 |

The server answers `loaded in 180.1s` and I have to leave it alone for three minutes. Not three minutes of nothing: during the load `MemAvailable` falls to about 36 GiB, and whatever else was using the GPU has to be gone before the unit starts. I host Qwen3.8-Flash-Next, NVIDIA’s quantisation-aware-distilled FP4 checkpoint (`nvidia/Qwen3.8-Flash-Next-FP4-Local`, revision `b9ce18a6`), through [TensorFold](https://github.com/ashhart/TensorFold) v1.1.4’s Zig engine on one DGX Spark: GB10, 128 GB [Unified memory](https://gurkanozkan.ai/concepts/unified-memory/) shared with the host, `free -g` reporting 121 GB of it to Linux. The recipe is Mia’s AI Lab’s public single-Spark recipe, patches `0001`-`0017` on the stock engine. {#p-ys87bv}

The boot line from `docker logs qwen38-flash-next-tf` says exactly what the machine is spending: {#p-fphatc}

```plaintext
[tensorfold] CUDA sm_121 device 0 (NVIDIA GB10, memory shared with the host):
model 75.92 GiB; 4 streams at once, 4.78 GiB each at a 262144-token window
(fp8 caches), of 25.9 GiB left after a 10.0 GiB reserve; prompts in passes of
up to 3072 rows, a 6.7 GiB prompt cache; own glue kernels
```

That one line is the whole budget, and everything below is arithmetic on it. {#p-iyz0g8}

## The budget {#the-budget}

The [Quantisation](https://gurkanozkan.ai/concepts/quantization/)-aware-distilled FP4 weights take 75.92 GiB. The container runs with `TENSORFOLD_MEMORY_RESERVE_GIB=10`, so the engine keeps a 10.00 GiB (10 GiB) reserve of `MemAvailable` for the operating system. What remains after weights and reserve, 25.9 GiB, is the [KV cache](https://gurkanozkan.ai/concepts/kv-cache/) pool. `--parallel 4` with `--context 262144` and `--kv-cache fp8` gives each stream a full 262,144-token window, 4.78 GiB each: four windows come to 19.12 GiB, and 25.9 minus 19.12 leaves the 6.7 GiB (6.70 to two decimals) that the line calls the prompt cache. {#p-w3e8xj}

Two details sit behind those numbers. The checkpoint on disk is about 96 GiB and loading takes about 81 GiB of memory, because the 26.8 GiB n-gram table is read from disk through the page cache instead of being held on the GPU: the recipe measured `MemAvailable` falling from 117.3 to 36.3 GiB at load with the vision helper on. My boot lands in the same place, which is why nothing else can be loading when the unit starts. {#p-e62m93}

And the fp8 cache is what makes the arithmetic close at all: 17,888 bytes a token against 30,784 for bf16, per the recipe’s `scripts/config.sh`. The engine’s own table prices one fp8 window at 4.37 GiB; the boot line charges 4.78, and I did not isolate what the extra 0.41 GiB a stream covers. fp8 is also not free in quality: the recipe’s check puts it at about 98.8% top-1 agreement with a bf16 cache, so a free-running reply can differ from the exact one. I did not A/B the two caches on my own workload. {#p-hden44}

The prompt cache is a cap, not a reservation, as the recipe’s README puts it: it takes at most what the weights and the streams leave. Passing `--prompt-cache-gib 12` changes nothing here. The 6.7 GiB exists because I run four streams and not five: one fewer window is one more window’s 4.78 GiB of prompt cache, which is why this build sits at 4. {#p-u39wob}

## Why four and not six {#why-four-and-not-six}

The engine admits each stream against the pool at start. The recipe measured where the wall is, and it is not far from where I sit: `PARALLEL=8` at 262,144 tokens needs 38.27 GiB of cache and the engine answered “the CUDA memory budget leaves 27.22 GiB”. Six streams at 28.7 GiB fit only with a 7 GiB reserve, and the recipe’s run of that configuration kept its speed and exactness checks but left 0.2 GiB of prompt cache, where `tools/prompt_reuse.py 30000` failed: a 32k-token follow-up resumed 4.7% of its prompt and prefilled again in 13.8 s. A server that re-reads every multi-turn history on every turn is not serving agents; it is re-reading them. Cutting the reserve is not free either: each GiB the reserve gives up is a GiB of page cache the n-gram table loses. {#p-3nd6e8}

**Chart:** Eight windows need 38.27 GiB against a budget of 27.22; six fit only by leaving the prompt cache almost empty. Source: the recipe's own measurements (PARALLEL=8, PARALLEL=6) and my arithmetic on the boot line.

*Cache the stream windows need at 262,144 tokens, in GiB*

| Streams | GiB for the windows |
| ---: | ---: |
| 4 | 19.12 |
| 6 | 28.7 |
| 8 | 38.27 |

Source: the recipe's own measurements and my arithmetic on the boot line

So the concurrency cap is not a policy I chose. It is what is left after the weights. The recipe’s own default is `PARALLEL=5` at a 9 GiB reserve, and I run 4 at 10: one fewer stream bought a bigger reserve and, through issue #81’s arithmetic, one window’s worth of prompt cache. {#p-chty0i}

## The other thing that has to stop {#the-other-thing-that-has-to-stop}

The requirement in the recipe is “a DGX Spark with nothing else large on the GPU”, and the default window wants `MemAvailable` of about 107 GiB at start. In October, when I moved this endpoint from an earlier engine, the old server was stopped and its supervisor and timers disabled before TensorFold took the loopback port, so nothing could restart beside it. There is no co-tenancy on this box: another GPU model running during my load takes the memory the reserve is supposed to protect, and a server started on a shrunk pool carries that shrink for its whole life, since the pool is sized once, from `MemAvailable` after the weights land. {#p-ccxs0z}

## What the 6.7 GiB buys, and when it leaks {#what-the-67-gib-buys-and-when-it-leaks}

`curl -s localhost:8888/metrics` on the running server reports 247,711,917 prompt tokens asked and 192,893,509 restored from kept prompt states, 77.9% of them (my division, read at about 19:55 on 14 November; the counters move with every request). The `done` lines in the log carry `prompt=` and `cached=` per request, so I can split that by prompt length over 13,638 requests from just under three days of agent traffic. {#p-n7xl8e}

Short requests, 2k-20k tokens: 8,676 of them, and the mean share of the prompt resumed from cache is 0.438, median 0.558. A short turn is served from cache about half the time, and when it is not, it prefills from scratch, which costs little. {#p-y7h8uw}

Long conversations, 160k tokens and up: 123 requests, mean share 0.738, median 0.997, but the distribution is two lumps, not a hill. 89 resumed above 95% of their prompt. 8 resumed nothing at all: prompts of 160,219 to 166,784 tokens that found nothing in the 6.7 GiB and prefilled all of it. A conversation that outgrows the cache does not merely get less help; it loses the whole prefix, every turn, while a short one keeps getting served. {#p-qxg08n}

**Chart:** A long request that fell out of the prompt cache waited 185.1 s for its first token, against 5.6 s for one that stayed in. Source: the per-request done lines of the server log, just under three days of traffic, 14 November 2026.

*Time to first token of the 160k-token requests, in seconds*

| Share of the prompt resumed from cache | Requests | Mean TTFT (s) | Median TTFT (s) |
| --- | ---: | ---: | ---: |
| Resumed above 95% | 89 | 5.6 | 0.99 |
| Resumed nothing | 8 | 185.1 | 174.1 |

The cost is in time to first token. A 160k+ request that hit the cache waited 5.6 s on average over 89 requests, median 0.99 s; a handful of slow warm requests pull the mean up, and I did not isolate why. One that fell out waited 185.1 s, median 174.1 s, over 8 requests, every one of them between 137.4 s and 274.5 s. A cold long turn is worth half a boot. The cumulative counter agrees with the leak: the `prompt cache:` lines show hits 12,941 against evicted 14,701, states evicted more often than served. {#p-4g180n}

What I have not isolated is what evicts them. The log does not say which workload drove the 160k+ prompts, and the 8 cold requests could not be traced to a specific client from the record I read. The mechanism I can point at is arithmetic: at the fp8 price of 17,888 bytes a token, a 181,271-token prompt, which is also in the log, is about 3.02 GiB of the 6.7 GiB kept, so two such prefixes at once nearly fill it, and the least-used state goes. Four streams each holding a long window and a prompt cache that fits two long conversations is the same pool cut twice. {#p-pz9erb}

**Figure:** The cache fits two long conversations; a third pushes the least recently used one out. Source: the server's own boot line (docker logs) and my arithmetic on it; the per-token cost is from the recipe's scripts/config.sh.

The 25.9 GiB pool is four stream windows of 4.78 GiB and the 6.7 GiB prompt cache. A conversation of 181,271 tokens costs about 3.02 GiB of cache at 17,888 bytes a token, so two of them nearly fill it.

Two knobs move it and both cost the thing that makes the box useful. `PARALLEL=3` leaves a second window’s 4.78 GiB to the cache, as the recipe’s issue #81 notes for its own build. A shorter `--context` leaves more still: the recipe calculates eight streams at 163,840 tokens at about 24 GiB of cache, more streams than I can run now, though it says the calculation was never run. {#p-eyfwyo}

The open question is whether long agent conversations deserve their own slot count, or whether the answer is to keep histories shorter than the cache instead of buying capacity for them. I have run the server for just under three days with one configuration. The 185 s waits are real and the fix is untested. {#p-xde1yr}

*Numbers from one machine and one server process running since 23:22 on 11 November. The PARALLEL=8 and PARALLEL=6 measurements are Mia’s AI Lab’s, on their Spark.* {#p-ajgyza}