Skip to content On the map On the map: KV cache uses related to KV cache — The attention keys and values of the tokens a transformer has already processed, kept in memory so that they are not recomputed for every new token it generates. KV cache Inference serving — Running a trained model as a service that accepts requests and returns outputs, scheduling computation and memory so that many requests can be answered with acceptable latency and throughput. Inference serving Prefix caching — Reusing the key-value cache already computed for a prompt prefix, such as a shared system prompt or the earlier turns of a conversation, for later requests that begin with the same tokens. Prefix caching State-space models — Sequence models, such as Mamba, that carry a fixed-size recurrent state through a sequence instead of attending to every earlier token, so that their memory per sequence does not grow with context length. State-space models Speculative decoding on GB10: a wider draft speeds up one stream and slows sixteen — Choose the speculative-decoding draft width for the number of users you serve: in darkdatter's sweep on a DGX Spark, 16 draft tokens instead of 8 made one stream 28% faster and sixteen streams 10% slower. Speculative decoding on GB10 What 128 GB of unified memory buys you: serving LLMs on a DGX Spark — On a GB10 the GPU memory fraction is a share of the whole machine: size the KV cache from the workload, and expect bandwidth and prefill, not capacity, to set the limits. What 128 GB of unified memory buys you Show on the main map →