Qwen3.8 on RTX PRO 6000 Blackwell: Faster Long-Context Agent Tasks
Kernels, 4-bit quantization and speculative decoding for Qwen3.8 27B and Flash Next on one RTX PRO 6000 Blackwell: agent tasks in 36% less time and decode 2.1x on the 27B, Flash Next 1.8x on one or two GPUs, and the 374 GPU-hours it took.
28 October 2026. Benchmark results from a five-day optimization campaign, not a peer-reviewed paper.
Three setups, one goal: finish long-context agentic coding tasks sooner at the same quality. We tuned kernels, 4-bit quantization and speculative decoding for Qwen3.8 27B on one GPU and for Qwen3.8-Flash-Next on one or two GPUs.
Everything was measured from 24 to 28 October 2026 on NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs (AWS g7e Spot), with a single stream unless stated. The glossary at the end explains the short names. Hover over a chart for exact values, or open its data table. The repetition loops counted in the GPU-time chart are the subject of the previous post.
What changed
Agent task time, one change at a time
Mean time per run of W01: six long-context agentic coding tasks, with tool calls and grading included and engine start excluded. "17 of 18" counts graded task attempts that passed across all runs.
Lower is better ↓Each step's own comparison ran on one host; the ladder joins hosts, so read it as a description, not a controlled series. The last row is a projection, not a measured stack.
Data table
Decode speed as the stack grew
Output tokens per second for one stream, generating 2,048 tokens after a 64K, 128K or 256K-token prompt (three prompts pooled per length). Greedy decoding; speculative decoding keeps the output identical to plain greedy decoding.
Higher is better ↑Stock and the last point were measured on one host (28 October, three draws each); the points between come from earlier hosts. The first point used one prompt per length, and its raw files were not kept. Decode speed moves with how many drafted tokens are accepted, not only with kernel speed.
Data table
Where a run's time goes, and what each lever removes
One W01 run on the 27B stack before sparse prefill (mean of two runs, 680 s). Seconds per run and share of the run.
Data table
Decode speed against stock speculative decoding
For each setup, stock speculative decoding on the published checkpoint is 1.0×. The darker part of each bar is what this work added. A tick marks the same checkpoint with no speculative decoding.
Higher is better ↑For each setup, all three were measured on one host on 28 October, interleaved, each on a fresh engine start: three benchmark draws and one or two W01 runs each. Every best stack passes its quality gates (GSM8K, needle retrieval, W01 grade).
Data table
Serving several agents at once
Total output tokens per second when several long-prompt streams share one GPU: our stack before the INT4 draft head (solid) against stock FP8 + MTP5 (dashed). Each stream sends two requests of 2,048 output tokens; prefill time is included, so one stream shows less than its decode speed.
Higher is better ↑The two stacks ran on different hosts with the same serving limits. Stock keeps its KV cache in BF16, so it holds at most 8 streams at 64K and 5 at 128K; more streams wait in line and add no throughput. Our FP8 KV cache holds 1.74× the tokens, but at 128K and 16 streams it was full too: three requests were preempted and throughput fell. A 4-bit KV cache (next steps) holds 1.76× the tokens of FP8.
Data table
GPU time behind these results
GPU-hours by work item: GPUs per host × hours from launch to termination, including setup, idle time and hosts lost to Spot reclaims. All agent sessions that used hosts are included.
Data table
What could come next
Candidates ranked by expected effect on task time. Estimates are marked as such.
Terms and short names
How these numbers were measured
Hardware. NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB, SM120), on AWS g7e Spot instances: one GPU for the 27B and Flash Next 1-GPU setups, two GPUs with tensor parallelism for Flash Next. Speed was measured only on this GPU; the 32 GB RTX PRO 4500 was used for fit and quality checks only. No other heavy work ran during speed runs.
Serving. vLLM nightly with our patches. Decode speed: one stream, greedy, thinking on, 2,048 output tokens after real code-review prompts of 76.7K, 126.1K and 223.7K tokens, three prompts per length, pooled as (tokens − 1) ÷ decode time. Fresh engine start for each benchmark.
Quality gates. Every adopted stack keeps GSM8K (1,319 problems; 200 for early 27B rows) within its bar, passes 18 of 18 needle-retrieval probes at 64K–200K, and meets the W01 grade bar with no repetition loops. Speculative decoding with greedy verification is lossless; 4-bit weights, the 4-bit KV cache and sparse prefill change the numerics and are judged by these gates.
Caveats. The summary tiles and the setups card compare no speculation, stock and best on one host per setup (28 October reruns). The 27B task-time ladder, the 27B decode history and the concurrency card join hosts. W01 has six frozen tasks, so repeated runs are repeat episodes of the same tasks; trajectories differ between runs, and run-to-run spread is about ±5%.