Notes

In July I ran coding-agent evaluations on SWE-bench Verified against Qwen3.6-35B-A3B, served by vLLM 0.21 (nvcr.io/nvidia/vllm:26.05.post1-py3) on one DGX Spark, four requests at a time. The model is a Mixture of experts and a hybrid: three of every four layers are Gated DeltaNet linear attention, interleaved with full attention, and vLLM runs those layers through its Mamba state-cache code. On a 20-instance run, six instances ended at 1,801.6 to 1,801.9 seconds. With a LoRA adapter on the same base, 14 of 20 did.

The 1,802 seconds came from the client, not the model. The OpenAI Python client defaults to a 600-second timeout and two retries: three silent waits on a request stuck at 0 tokens per second, then an error. My harness scored each one as the model failing the task. For Model evaluation that is the real damage: a hung server looks exactly like a weak model. From then on the harness recorded early_stop and api_error per instance, so the result file can tell the two apart.

My first suspect was Prefix caching. The experimental “align” prefix-cache mode lives in that Mamba code path, and two vLLM issues fit: #42317, a 0% cache hit rate in align mode for this exact model (closed as fixed in June, after the build I was running), and #43587, still open, prefix caching failing for incremental multimodal requests on Mamba-attention hybrids. I added --no-enable-prefix-caching --enforce-eager, cut the client to timeout=240, max_retries=0 (a normal generation took about 90 seconds) and moved retries to the task: up to three attempts, each in a fresh sandbox. The first-turn hangs stopped. Stalls in the middle of a run did not.

Then the same configuration, on a server that had already served another workload, wedged on 15 of 15 instances, each ending at 721 to 727 seconds: three 240-second attempts.

What helped was procedure: a fresh server before every heavy run, and two short warm-up requests (“Write hello world.”, 40 tokens) before the real load. Two fresh, warmed-up runs on six of those fifteen instances, four at a time, finished without an API error. Those 12 instances are the only clean evidence I have under four-way load. Afterwards I ran 50, 63 and 50 instances one at a time, also with no API errors.

The last 50 needs a caveat. With the LoRA adapter the server still wedged after about 14 instances, even one request at a time. Its runner restarted the server after two consecutive API errors and deleted those rows before resuming, so that clean file cannot show the wedges in between.

Why the procedure works is still a hypothesis: state that builds up in a long-running server, plus Cold start effects in the linear-attention kernels and state caches. The evidence is a before-and-after comparison, 15 of 15 wedged on a used server against 0 of 12 on fresh ones. I never ran a fresh, warmed-up server without the two flags on this model, so I cannot say which part does the work.

I designed and directed these runs. An AI coding agent wrote most of the evaluation and retry code, ran many of the jobs and kept the project notes. Every figure above was re-checked against the saved result files in November 2026.