---
title: "Memory is architecture: KV-cache footprints of 2,068 open models"
kind: essay
date: 2026-11-14
author: Gürkan Özkan
url: https://gurkanozkan.ai/writing/kv-cache-census/
markdown: https://gurkanozkan.ai/writing/kv-cache-census.md
description: Across a census of 2,331 open-weight checkpoints, parameter count explains 0.398 of the variance in KV-cache bytes per token while attention geometry explains 0.792, and the median model that fits its weights on a 128 GiB box holds 30 sequences at 32k context and 8 at 128k.
topics:
  - ai
  - analytics
tags:
  - kv-cache
  - serving
  - census
  - gb10
  - attention
  - regression
concepts:
  - inference-serving
  - kv-cache
  - mixture-of-experts
  - pre-registration
  - quantization
  - unified-memory
---

# Memory is architecture: KV-cache footprints of 2,068 open models

**Chart:** Cover: The same 32,768 tokens cost 4.83 GB of cache on Qwen3-8B and 2.30 GB on a model 125 times its size. Source: my census of open-weight configs on the Hugging Face Hub (analysis/kv.py, BF16 accounting).

*KV cache of one 32,768-token sequence in BF16, in GB*

| Model | GB per 32k sequence |
| --- | ---: |
| Qwen3-8B | 4.83 |
| Qwen3-30B-A3B | 3.22 |
| Kimi-K2-Base | 2.30 |

Qwen3-8B spends 4.83 GB of cache on one 32,768-token sequence in BF16. Qwen3-30B-A3B spends 3.22 GB, and Kimi-K2-Base, 125 times its size, spends 2.30 GB. A bigger model can need less memory for the same conversation. I ran the arithmetic over every open-weight config I could download, and the inversion is the normal case, not the exception. {#p-znjqjq}

The census: 2,331 checkpoint rows from 33 original publisher organisations on the Hugging Face Hub, released between 2023-01-15 and 2026-10-09 (the Hub createdAt day stands in for the release date, a limitation I flag rather than fix). Only original publisher organisations are in: fine-tunes, merges and quantised re-uploads from third-party accounts stay out, because their configs inherit geometry and their parameter counts drift. Collapsing base/instruct twins with identical geometry leaves 2,068 family-dedup rows. 237 gated repositories were skipped because their config.json is not publicly downloadable and I never use a Hub token; that list is 130 google, 42 meta-llama and 24 CohereLabs entries, so two of the most-cited families are absent wholesale. Nine configs failed to parse and were excluded rather than guessed: two SmolVLM2 checkpoints missing num_attention_heads, DeepSeek-V4.1-Flash with an unknown compress_ratios entry, three vision wrappers whose text-backbone fields were missing, two Hymba checkpoints with an unregistered model_type, and one VersaViT without num_hidden_layers. 2,437 config.json snapshots sit under data/raw/configs/, and parameter counts come from safetensors metadata where it exists (1,964 rows), from the config where it does not (263), and are absent for 104 rows. {#p-jw4y45}

## What the accounting does, and what it assumes {#what-the-accounting-does-and-what-it-assumes}

Every headline number here assumes BF16: 2 bytes per element, cache and weights alike. I did not fold FP8 into these figures. {#p-zmyuph}

The accounting rules are code, covered by 107 unit tests. A full attention layer caches 2 × KV heads × head_dim × 2 bytes per token. A sliding-window layer holds min(T, W) tokens, so its marginal cost at long contexts is zero. An MLA layer caches the latent, kv_lora_rank + qk_rope_head_dim elements per token: DeepSeek-V2 ships 512 + 64 = 576, so its 60 latent layers cost 69,120 bytes per token, where 128 heads at head_dim 192 in plain attention would cost about 5.9 MB. A Mamba, Gated DeltaNet, lightning or RWKV layer holds a constant state computed from the config, independent of T, counted once per sequence. Cross-layer shared caches are counted once. The label is bare mha, gqa or mqa only when the identity 2 × layers × KV heads × head_dim × 2 holds exactly; anything windowed, hybrid, latent or shared gets a suffix, so the 53 MLA rows, 162 hybrid rows and 55 windowed rows in the dedup set are visible as what they are. Unknown architectures raise an error, never a silent guess; that is where the nine rows above went. {#p-ucrwal}

**Figure:** The accounting rules behind every number here: four kinds of layer, four ways the cache grows with context. Source: my census of open-weight configs on the Hugging Face Hub (analysis/kv.py, BF16 accounting).

A full attention layer caches 2 × KV heads × head_dim × 2 bytes for every token. A sliding-window layer keeps min(T, W) tokens, so past the window a new token adds nothing. A latent (MLA) layer caches kv_lora_rank plus the rope dimension per token, 576 elements in DeepSeek-V2. A recurrent layer holds a constant state per sequence.

These are theoretical minima, not allocations. The validation experiment instantiates seven models from real configs, reduced depth, random init, on CPU, and reads the cache tensors HF transformers actually allocates: 6 of 7 rows match the analytic figure within 1%. The set spans the classes: microsoft/phi-2 and Qwen2.5-0.5B for plain attention, starcoder2-3b and gpt-oss-20b for windowed, DeepSeek-V2-Lite for MLA, granite-4.0-h-micro for the hybrid, mamba-2.8b pure recurrent. The largest gap is 5.26% on pure Mamba-2.8b, whose conv state the analytic figure undercounts, kept as a finding rather than smoothed away. Two implementation habits the validation caught: transformers truncates a sliding-window cache to W − 1 positions, and HF stores the MLA latent compressed, not decompressed, so the theoretical latent is close to what ships. The plan behind the statistics was written down before the trends were computed, thresholds included, in the [Pre-registration](https://gurkanozkan.ai/concepts/pre-registration/) sense; the plan file also logs the deviations made afterwards. {#p-i94za7}

## Parameter count is a weak predictor {#parameter-count-is-a-weak-predictor}

I regressed log10 KV bytes per token on log10 total parameters over the 1,930 dedup rows that have both a parameter count and a growing cache (97 rows have no parameter count; 67 hold a pure recurrent design whose marginal cost per token is zero: RWKV and friends). The fit is a 0.39 slope with R² = 0.398; the bootstrap 95% interval for that R² runs from 0.350 to 0.448 over 1,000 resamples, seed 0. Size explains under half the variance, and the residual spread runs from 32 bytes per token (a toy GPTNeoX) to 2,621,440 (a 65B-era MHA model: 80 layers, 64 KV heads, head_dim 128, no sharing). {#p-azloix}

Swap in the geometry. Standardised predictors, on the same 1,930 rows: GQA ratio log(1 + query heads / KV heads), head_dim, depth, window-layer share, recurrent-layer share, an MLA-latent indicator, a cross-layer-sharing indicator. Without parameters that model reaches R² = 0.792; with parameters added, 0.841. Parameters contribute 0.049 on top, once the shape of attention is known. {#p-p7j4sv}

**Chart:** Geometry and mechanisms explain twice what parameter count alone does; parameters add 0.049 on top. Source: my census of open-weight configs on the Hugging Face Hub (analysis/kv.py, BF16 accounting); ordinary least squares.

*Variance in log10 KV bytes per token explained (R squared), 1,930 rows*

| Predictors | R squared |
| --- | ---: |
| Parameters only | 0.398 |
| Geometry and mechanisms | 0.792 |
| Both | 0.841 |

Source: my census of open-weight configs on the Hugging Face Hub (analysis/kv.py, BF16 accounting)

Two honesty notes about that 0.792. First, the mechanism set is not raw geometry alone: the window, recurrent and latent shares are architectural mechanisms, not dimensions. Layers, KV heads and head_dim by themselves reach R² = 0.534 on the same rows, still well above the 0.398, but the strong number needs the mechanisms. Second, the sample leans 2025-heavy: 243 rows from 2023, 563 from 2024, 949 from 2025, 313 from 2026. {#p-it6cg1}

## Twenty-six times the bill for 2.4 times the weights {#twenty-six-times-the-bill-for-24-times-the-weights}

Qwen3-8B and Qwen3-4B-Base share the geometry that matters: 36 layers, 8 KV heads, head_dim 128, GQA. Same 147,456 bytes per token, same 4,831,838,208 bytes for one 32k sequence. Qwen-72B, from the pre-GQA generation, caches 2,621,440 bytes per token, 85.90 GB per 32k sequence: 26.7 times what Qwen3-30B-A3B spends per token, on top of a weight footprint 2.4 times its size (72B against 30B parameters). {#p-c533tj}

[MoE](https://gurkanozkan.ai/concepts/mixture-of-experts/) helps per parameter, not per sequence: Qwen3-30B-A3B reads about 3B weights per token but all 30B must stay resident, and its cache bill per token is set by 48 layers and 4 KV heads whether the experts fire or not. Across the census, the median MoE row spends 1,110 bytes per token per billion parameters against 62,941 for a dense model, a Mann-Whitney p of 10 to the minus 99. Per sequence, though, nothing changes; the saving is bandwidth, which one shared pool on a GB10 makes the other wall. This piece is only about the capacity wall. {#p-9oyr5o}

## On a 128 GiB box {#on-a-128-gib-box}

For each of the 1,971 dedup rows with a parameter count I subtracted BF16 weights and a 6 GiB runtime allowance from 128 GiB, then divided what is left by the per-sequence cache at the two context lengths that matter. Recurrent state enters each sequence’s bytes; window layers saturate at min(T, W); weights that alone exceed the budget score zero. {#p-rap4j3}

Counting zeros, the median model holds 27 concurrent sequences at 32k context and 6 at 128k. 169 rows cannot fit their own BF16 weights plus overhead, and 8.9% hold zero sequences at 32k, 12.6% at 128k. Counting only rows that fit at all, the headline the study uses, the one the cover draws: 30 sequences at 32k context, 8 at 128k context. I state both because the difference between 30 and 27 is entirely the models that do not load. {#p-bj4ih8}

**Chart:** Quadrupling the context cuts the median batch from 30 sequences to 8. Source: my census of open-weight configs on the Hugging Face Hub (analysis/kv.py, BF16 accounting); pool = 128 GiB minus BF16 weights minus 6 GiB overhead.

*Median concurrent sequences on a 128 GiB box, models that fit their weights*

| Context | Sequences |
| --- | ---: |
| 32k context | 30 |
| 128k context | 8 |

Source: my census of open-weight configs on the Hugging Face Hub (analysis/kv.py, BF16 accounting); pool = 128 GiB minus BF16 weights minus 6 GiB runtime allowance

Worked: Qwen3-8B’s BF16 weights take 15.3 GiB, the pool is 106.7 GiB, four 32k sequences cost 18.0 GiB. gpt-oss-120b, whose 36,864-byte marginal comes from half its layers windowed at W = 128, still cannot load its BF16 weights into 128 GiB at all; its 1.13 GB at 32k is a moot quantity until you quantise the weights. That is the practical shape of the result: [Quantisation](https://gurkanozkan.ai/concepts/quantization/) moves you between “zero” and “median”, while the architecture sets what the median even is. {#p-q1pre0}

## What to do with it {#what-to-do-with-it}

Before choosing a quantisation or a second GPU, read four numbers out of the model’s config.json: layers, KV heads, head_dim, and what fraction of layers are windowed, recurrent or latent. Qwen3-4B-Base costs the same 4.5 GiB per 32k sequence as Qwen3-8B because they share 36 layers, 8 KV heads and head_dim 128. A spread this wide inside one census means parameter count is a proxy that fails exactly when serving gets expensive: long contexts and many sequences, the regime where the [cache](https://gurkanozkan.ai/concepts/kv-cache/), not the weights, fills the machine. The open question the census cannot answer is the served one: the analytic numbers are theoretical minima, and the seven-model validation, while reassuring on the full-attention rows, left the hybrid and MLA implementations allocating their own way. How far a real engine drifts from these numbers under load, I have not measured here. {#p-lvg5li}