Fifteen ways my coding agent and I mis-measured two models on one DGX Spark

In July I wanted to know whether poolside’s Laguna-S-2.1 should replace Qwen3.6-35B-A3B, my daily coding model on my DGX Spark since 5 July. An AI coding agent, Claude Code, did the hands-on work: it wrote the harnesses, ran the server and the jobs, did the statistics and drafted the reports. “We” below means the agent and me. A ten-problem smoke test on 23 July set the starting belief: Qwen for speed, about 99 tokens per second to Laguna’s 40; Laguna for quality, 10 of 10 to Qwen’s 9.

Between 23 and 27 July we used about 60 hours of machine time, by the agent’s own estimate, and got an answer with a p-value under each half of it. On the way we measured it wrong fifteen times, and several of the wrong measurements looked like findings.

The machine is a GB10: Grace-Blackwell, ARM64, 128 GB of memory shared by CPU and GPU, roughly 273 GB/s of bandwidth. Both models ran on vLLM 0.25.1, quantised to 4-bit NVFP4:

• Laguna-S-2.1 from poolside: a mixture-of-experts model, 117.6B parameters in total and 8.5B active per token, 67 GB of weights.
• Qwen3.6-35B-A3B from the Qwen team at Alibaba: 35B in total, 3B active, with linear-attention (Gated DeltaNet) layers interleaved with full attention. We tested unsloth’s third-party NVFP4-Fast build (24 GB), not Alibaba’s own release, and every Qwen number here belongs to that build on this stack.

Everything tied

The first plan was HumanEval, compare, done. Laguna scored 94.5% and Qwen 95.1% on its 164 problems (Fisher’s exact p = 1.000). MBPP-sanitized, 427 problems: 88.5% against 88.1%. Then we wrote 66 problems of our own, algorithms and bug fixes graded against a brute-force reference on 40 random inputs. Both models solved 65.

On none of the three did they differ by more than two problems: saturated benchmarks cannot rank two models at the ceiling. So we moved to LiveCodeBench, 25 AtCoder problems from January to April 2025 (picked by contest date to limit contamination), and BigCodeBench-Hard, whose tasks call real libraries and are checked by unit tests.

1. Greedy decoding broke one model and not the other

Our early runs used temperature 0, greedy decoding (always take the most likely next token), with top_p 0.95. Laguna, thinking on, the same 12 hard LiveCodeBench problems:

The pass count barely moved. The cost did, and the reason only showed in the raw output. On one problem the model wrote two sentences over and over until the budget ran out, 1,813 and 1,810 times:

Alternatively, perhaps think of the expression as (A_i + C_k) * B_j + C_k * A_i. Hmm, but I’m not sure.

The output started well, restating the problem correctly and noticing the constraints. Then it fell into a loop, and greedy decoding cannot leave one: once a repeat is the most likely continuation, it stays the most likely continuation. Holtzman and colleagues described this degeneration in 2019; it is why nucleus sampling exists. Of the 1.2M greedy tokens, 1,152,000 came from the nine runs that each burned the whole 128,000-token budget.

A crude loop score, the share of repeated sentences longer than 40 characters, catches it automatically:

def loop_ratio(t):
    s = [x.strip() for x in re.split(r'(?<=[.?!])\s+', t) if len(x.strip()) > 40]
    return 0.0 if len(s) < 5 else 100 * (1 - len(Counter(s)) / len(s))

Under greedy decoding it split passes (0.8–5.4%) from failures (79.6–99.2%) with nothing in between. Cutting generation at 50% repetition would have saved roughly 1.15M tokens and more than five hours without losing a correct answer; that is an estimate from the logs, and we never ran the rule. At temperature 0.7 the split blurs (one passing output reaches 13.8%) and looping drops without disappearing: one Laguna attempt on BigCodeBench/341 ran to its 32,000-token cap at 90.8% repetition.

Qwen barely looped under greedy decoding (0.3–10.2%) and solved 8 of the same 12 with 713k tokens in 136 minutes. We moved both to temperature 0.7, top_p 1.0, top_k 20, with two caveats: three parameters changed at once, so the table compares two settings rather than two temperatures; and those values are poolside’s recommendation for Laguna, while Qwen’s model card recommends different ones.

2. A token budget that stopped the model mid-thought

Qwen on LiveCodeBench, thinking on, max_tokens at 20,000: 2 of 24 solved, 8.3%, with 22 outputs cut off at the limit. At 64,000: 13 of 16, 81.2%, one cut off.

The two sets mixed medium and hard problems differently, so the fair comparison is the 16 problems both runs share: 1 solved at 20k, 13 at 64k. On those problems the model had been stopped mid-reasoning, and the grader scored each unfinished thought as a wrong answer.

We had confused two knobs. The context window (256K tokens for both models) is how much the model can hold; max_tokens is how much it may write in one reply. The final runs used 128,000 on LiveCodeBench and 32,000 on BigCodeBench, with truncations counted in every run.

The 81.2% is close to the 80.4 that Qwen reports for LiveCodeBench v6. The problems differ, so that is consistent with the NVFP4 build losing little quality, not proof of it.

3 and 4. One run, and no statistics

On BigCodeBench-Hard we first ran 25 tasks once per model. Laguna passed 11 (44%), Qwen 7 (28%). Sixteen points, and it went into the report as a finding.

Then came the test that should have come first. Fisher’s exact test on 11/25 against 7/25 gives p = 0.377; the 95% Wilson intervals, 26.7–62.9% and 14.3–47.6%, overlap almost entirely. LiveCodeBench had the same problem: Qwen’s single-run lead of 84% to 56% gave p = 0.062.

When we extended the run to 50 tasks, Laguna passed 8 of the original 25, 32%, with the same model, tasks and configuration. Sampling alone moved it 12 points, against a 16-point “finding”. Those first 25 were also easier than the rest, because the script took tasks in id order: over the later four runs Laguna passed 34 of 100 attempts on the first 25 tasks and 22 of 100 on the next 25, Qwen 24 and 12.

The fix was dull: 50 tasks, four runs per model, 200 attempts each, nothing changed between runs.

Laguna passed 14, 16, 11 and 15 of 50 (28.0% ± 4.3 across runs), Qwen 7, 10, 10 and 9 (18.0% ± 2.8). Fisher’s exact p = 0.024, Welch’s t = 3.87. Had we stopped after any single run, we would have reported a gap of 14, 12, 2 or 12 points; after three, p was still 0.073. How many runs a gap needs is a statistical power question best asked before the runs: here a ten-point gap needed four runs of 50 tasks, while the 24-point LiveCodeBench gap reached p = 0.016 with two runs of 25.

A pooled Fisher test treats the 200 attempts as independent, which they are not: each task appears four times. A recount per task, added in a review this November, gives Laguna more passing runs on 13 tasks, Qwen on 3, and 34 tied; a sign test gives p ≈ 0.021, a bootstrap 95% interval for the gap +3.0 to +17.5 points. The result holds, with a wide interval.

5 and 6. A parser that ate the output, and no raw text to notice

The first runs went through vLLM’s chat endpoint with --reasoning-parser poolside_v1, and we kept only pass or fail. reasoning_content was always empty, and when the model hit the token limit before closing its </think> block, content came back empty too: 8,000 tokens generated, 0 characters returned. A benchmark scores that as a wrong answer.

As far as the agent traced it, the parser inherits a default that assumes thinking is off while Laguna’s template assumes it is on, and it decides once, at server start, so a per-request enable_thinking: true changes nothing. A local patch that re-based it on DeepSeekV3ReasoningWithThinkingParser, which 0.25.1 already ships, fixed normal replies; truncated ones still lost their text. The bug is vLLM issue #49379, opened on 21 July and still open in mid-November; the proposed fix, PR #49434, was closed without being merged.

The lasting fix was to bypass the parser: raw requests to /v1/completions, each model’s own chat template applied with the tokeniser’s apply_chat_template (never by hand), and every completion written to disk beside a per-task CSV row (pass, tokens, seconds, truncated, has_code, loop_pct).

Not saving that text was the sixth mistake. Without it, a low score could mean bad reasoning, a loop or an empty reply, and nobody could tell which; the 1,813 repetitions only became visible once the text was on disk.

7. A harness that was wrong without an error

We first tried SWE-bench Verified with an evaluation harness written for another project. It hard-coded that project’s training template: ChatML with <think> padding, a stop string of <|im_end|>, and max_tokens fixed at 4,096. Nothing failed. It returned 36% for the base Qwen model, which Qwen reports at 73.4. On Laguna, which uses a different template, it would have broken outright.

Some gap between scaffolds is normal; half the published score is not. We discarded the result and switched to mini-swe-agent, the minimal harness from the SWE-agent team. The rule we wrote down afterwards: before trusting an inherited evaluation harness, grep it for template, max_tokens, stop= and tools.

The SWE-bench run stopped after 28 Laguna instances and is not in the results. Each instance took about 24 minutes of wall clock with three running in parallel (a single instance often took over an hour), so the 100-instance slice we had started would have taken about 40 hours per model. Our LiveCodeBench and BigCodeBench numbers come from short scripts the agent wrote. They apply each model’s own template and run each benchmark’s own tests, but they are local measurements on subsets, not leaderboard numbers.

8 to 15. The operational ones

These happened in the job-running layer the agent operated. Most cost hours; a few also corrupted results until caught.

• Docker on ARM. The agent assumed SWE-bench’s x86 containers could not run on ARM64. docker run --privileged --rm tonistiigi/binfmt --install amd64 proved otherwise.
• Zombie workers. Killing the vLLM parent left its EngineCore workers holding 80 GB of unified memory, and the next server died on start. Kill by PID, then check free -g.
• Readiness. /v1/models answers before the model can generate; three runs started too early and came back empty.
• The grader’s Python. BigCodeBench grading ran under the system Python, which lacked pandas, so tasks failed on import and counted against the model.
• Workers changed mid-run. Going from 3 to 6 parallel workers halfway through mixed two configurations in one measurement and caused timeouts.
• The wrong PID. Runs started with setsid were tracked by the wrapper’s PID, not the Python process’s. Wait on an end marker in the log instead.
• pgrep matched itself. A monitor’s pattern matched its own command line and raised a false “process died” alarm.
• Not reading its own notes. The agent’s memory already held a note from two weeks earlier on running SWE-bench containers on this ARM machine, and it worked the problem out again from scratch.

The protocol, and what it found

The protocol we ended with is also what I would hand a team choosing a coding model for its own hardware:

1. Preflight: run every task’s reference solution through the grader and drop tasks that fail (BigCodeBench-Hard went from 148 tasks to 50 after library and reference checks).
2. Check the server with a real generation request.
3. Warm it up with two simple sequential requests.
4. Sample at temperature 0.7, never greedy.
5. Give reasoning tasks max_tokens of at least 64,000.
6. Use the benchmark’s official harness where one exists.
7. Save the raw output of every task.
8. Run three or four times with the configuration frozen.
9. Sort failures into truncation, environment and real.
10. Report Fisher’s exact test, Wilson intervals and run-to-run variance.

The top three rows predate every fix: single greedy runs through the chat endpoint, never repeated. They stay because both models sit at the ceiling on large sets, not because they meet the protocol.

The bottom two rows point in opposite directions. Qwen leads by 24 points on LiveCodeBench, and the gap widens with difficulty: on hard problems it solved 18 of 24 attempts to Laguna’s 9, on medium ones 23 of 26 to 20. Laguna leads by 10 points on BigCodeBench-Hard and uses fewer tokens doing it: 2,606 per attempt against 5,061, and 9,306 per solved task against 28,117.

Speed and memory favour Qwen on this machine: about 99 tokens per second in a single stream against 40, 212.5 against 82.8 over four concurrent requests, 24 GB of weights against 67.

poolside’s launch post reports agentic benchmarks only; Qwen’s reports LiveCodeBench but not BigCodeBench. Our two measurements happen to land in exactly those gaps. With one pair of models I can’t say whether that is a pattern or a coincidence.

The agent’s report closed with a decision guide: Qwen-Fast as the general daily model, Laguna for library-heavy and agentic work, the agentic part resting on poolside’s published scores rather than ours. That is a July 2026 snapshot. My local serving has since moved to a newer model, Qwen3.8-Flash-Next, which none of these runs tested. The protocol above is the part that carries over.

The sixteenth mistake

The numbers in this essay are the corrected ones. The agent corrected its July reports as the work went, yet they still carried stale figures and one claim the code does not support:

• The summary report’s decision guide recommends Qwen for algorithmic work with “84% vs 56%” and Laguna for library work with “44% vs 28%”, the single-run numbers the same report had already replaced. Both per-model result files, in sections headed final, still give Qwen a 28-point LiveCodeBench lead. It is 24.
• “2.6× fewer tokens” for Laguna on BigCodeBench comes from the 25-task run and survived into the main report’s decision guide. On the 50 × 4 data it is 1.9×.
• The main report lists a deterministic shuffle among its harness techniques, to avoid first-N bias. The BigCodeBench script sorts tasks by id and takes the first 50, the ordering that inflated the 25-task scores.

Each fix landed in the file being edited; every file that had quoted the old number kept it.

An AI coding agent (Claude Code) wrote the harnesses, ran the server and the jobs, did the July statistics and drafted the reports. In November an AI review pass re-checked the numbers against the CSVs and logs and added the task-level sign test and bootstrap.