Skip to content On the map On the map: Prefix caching uses Prefix caching — Reusing the key-value cache already computed for a prompt prefix, such as a shared system prompt or the earlier turns of a conversation, for later requests that begin with the same tokens. Prefix caching Inference serving — Running a trained model as a service that accepts requests and returns outputs, scheduling computation and memory so that many requests can be answered with acceptable latency and throughput. Inference serving KV cache — The attention keys and values of the tokens a transformer has already processed, kept in memory so that they are not recomputed for every new token it generates. KV cache What 128 GB of unified memory buys you: serving LLMs on a DGX Spark — On a GB10 the GPU memory fraction is a share of the whole machine: size the KV cache from the workload, and expect bandwidth and prefill, not capacity, to set the limits. What 128 GB of unified memory buys you The 1,802-second hang: serving a hybrid linear-attention MoE under agent load — In July I ran coding-agent evaluations on SWE-bench Verified against Qwen3.6-35B-A3B, served by vLLM 0.21 ( ) on one DGX Spark, four requests at a time. The… The 1,802-second hang Show on the main map →