The Frankenstein, Part 1: DeepSeek-V4.1-Flash on Two RTX PRO 6000s

2 November 2026. A progress note from a running campaign. The measurements below are engine-level, and none of them is agent task time unless it says so. Not a peer-reviewed paper.

We’re building a server for DeepSeek-V4.1-Flash on a workstation with two RTX PRO 6000s, Frankenstein-style: other engines’ best parts first, then our own.

Part 1: the body, one part served (verify steps 2.07% shorter) and one up next (5.57 s less server time per task draw).

The target is DeepSeek-V4.1-Flash-1017 on one workstation: two RTX PRO 6000 Blackwell Workstation Edition GPUs, a 64-core Threadripper PRO 9985WX and 512 GB of RAM. The workload is long-context agentic coding, one request at a time, and the goal is to finish agent tasks sooner at the same graded quality.

The plan has two stages. First we combine the best parts of several existing engines. Then we replace parts with our own. Each part goes in only through a test whose decision rule we write before it runs.

Summary

• The body: vLLM with diffbot’s EXL3 3-bit pack. On 31 October it replaced our llama.cpp setup under a rule written before the run. Its total time for 18 agent-task episodes was 28.2% lower in that run, and it passed 13 of 18 against 15, within the rule’s limit.
• New part, served: an optional CPU-expert schedule from diffbot’s runtime. Each verify step took 2.07% less time, lower in 5 of 5 randomized blocks (p = 1/32). That is step time, not task time. Its quality test is running.
• New part, up next: a cache of token IDs per prompt region. It cut the server’s time outside queueing and prefill from 6.64–6.93 s to 1.16–1.23 s per task draw, 5.57 s per draw, in 8 randomized engine starts (p = 1/70), with identical prompt token IDs. It passed its registered test, but it is not task time and it is not adopted or served yet.

The body: vLLM with diffbot’s EXL3 3-bit pack

diffbot’s recipe serves DeepSeek-V4.1-Flash-1017 on two RTX PRO 6000s with vLLM at tensor-parallel 2:

• routed experts in 3-bit EXL3, turboderp’s format, calibrated with coolbho3k’s method;
• int4 Engram tables and a DSpark draft model for speculative decoding;
• about a third of the routed experts in host RAM, computed by an AVX-512 CPU runtime that diffbot wrote, so only activations and results cross PCIe.

We compared it with the setup we were serving: llama.cpp with Unsloth’s UD-IQ2_XXS GGUF, our own patches and a DSpark drafter. The decision rule, written before the run, adopted the vLLM engine if all three held:

1. its total time over 18 episodes of W01 is lower;
2. it passes at most 2 fewer of the 18 episodes;
3. its fidelity against a reference model’s output distribution is no worse.

W01 is our agent-task benchmark: six long-context coding and analysis tasks with real tool calls, graded mechanically. One draw runs all six, with thinking on, at temperature 1.0. Each engine ran three draws, each in a fresh prompt-cache namespace, timed on the client.

	vLLM + EXL3 3-bit	llama.cpp + UD-IQ2_XXS
W01 total, 18 episodes	1,067.18 s	1,486.61 s
Per draw	465.21 / 304.64 / 297.34 s	518.02 / 407.57 / 561.02 s
Episodes passed	13 of 18	15 of 18
Fidelity against a reference model’s output distribution: KL on three text sets, lower is closer	0.129 / 0.095 / 0.105	0.270 / 0.229 / 0.199

What this does and does not show:

• A lower total in this run, not a general speedup. This was a decision rule written before the run, not a randomized comparison. The draws ran in sequence on one afternoon: two llama.cpp draws, then the vLLM engine’s three, then the third llama.cpp draw. No significance test was registered, and the per-draw totals overlap.
• Passes: 13 against 15 is within the rule’s limit of 2 fewer. It is not evidence that quality is equal.
• Fidelity: KL against the output distribution of the checkpoint’s own reference code, teacher-forced on the top 64 log-probabilities, on three text sets. Lower is closer. Two runs of the reference itself differ by 0.017–0.031 on the same sets.
• Before schedule 2: this run measured the vLLM engine with its default CPU-expert schedule, before the part below was served, so its total and the step-time figure below don’t combine.

New part, served: diffbot’s second CPU-expert schedule

diffbot’s CPU runtime has an optional second schedule, EXL3_CPU_EXPERTS_SCHED=2, which splits each job’s work across the pool’s threads differently. Their README lists it as measured within noise on their 96-core CPU and not used. On our 64-core CPU, a screening run picked it out, and we then tested it against the default schedule.

The test (registered before launch): five blocks, each with one engine start per schedule in random order, replaying recorded W01 requests. The rule: schedule 2 is confirmed only if its time per verify step is lower in all five blocks.

Block	1	2	3	4	5	Mean
Time per verify step, schedule 2 against 1	−2.44%	−1.77%	−2.92%	−1.70%	−1.50%	−2.07%

It was lower in all five blocks: confirmed, one-sided exact sign test p = 1/32.

What this does and does not show:

• Step time, not task time. A verify step is one pass of the target model over the draft tokens. How much of a task’s time that is depends on the task.
• Quality is not settled. The schedule changes the order in which floating-point sums are added, so outputs can change. A fidelity check could not settle it: two starts of the same engine configuration already differed by more than the check’s tolerance.
• Served on a policy call. Our campaign manager put schedule 2 into serving on 1 November, on the step-time result, with a revert rule: a failure that only schedule 2 shows, a lower pass rate or a quality regression sends it back. A randomized W01 comparison of the two schedules, 10 blocks, is running now. Its result goes in the next note.

New part, up next: a prompt region cache

An agent request carries the whole conversation so far. In W01 the prompts are 105K–175K tokens long, and one text region between the chat template’s control tokens makes up 98–99% of each one. That region repeats verbatim from request to request, and the server tokenized it again every time: about 0.1–0.2 s per request on the CPU.

diffbot’s recipe already cuts each prompt at the template’s control tokens and tokenizes the regions in one batch call. Our hook wraps that call with a cache from region text to token IDs, bounded and least-recently-used first. A region seen before gets its stored IDs, and the rest are tokenized as before. Each region is tokenized on its own, with no padding, truncation or carry-over between regions, so stored IDs should equal fresh ones. We checked that on all 1,915 recorded W01 and agent-task requests, and again in every engine start below.

The test (registered before launch): eight engine starts, four with the cache and four without, in one of the 70 balanced orders, drawn at random. Each start replays three recorded W01 draws, 135 requests, one at a time. Each request asks for one output token, because the cache acts only before the engine runs. The measure is the server’s own time per request outside queueing and prefill (time to first token minus queue time minus prefill time, from vLLM’s metrics), summed per draw. That is chat-template rendering, tokenization and internal hand-offs.

Start (run order)	1	2	3	4	5	6	7	8
Cache	on	off	off	on	off	off	on	on
Seconds per W01 draw	1.212	6.640	6.710	1.173	6.770	6.927	1.159	1.234

Every start without the cache was slower than every start with it. The difference is 5.57 s per W01 draw; p = 1/70, the smallest this design can give. The prompt token IDs were identical in all eight starts. The six requests that start a fresh conversation, where the cache has nothing stored yet, were checked separately. With the cache on, they took 0.010 s longer in total; the registered cost check did not fire, which is not evidence of no cost.

What this does and does not show:

• Not task time. The replay is one request at a time with one output token. How much of the 5.57 s a live agent task saves is measured next, with the cache served.
• Not adopted or served yet. It is adopted only after the schedule comparison ends, and then served.
• What remains: with the cache on, the server still spent 0.75–2.01 s per draw outside queueing and prefill (the arm’s mean per draw).

Next

The next note will report the W01 comparison of the two schedules and the first live W01 runs with the region cache. After that, the parts list ranks the CPU-expert decode wait and the rest of the request front end next.

Credits

• DeepSeek for DeepSeek-V4.1-Flash-1017.
• diffbot for the EXL3 3-bit pack and its serving recipe, including the CPU-expert runtime with both schedules and the prompt segment encoder our cache wraps.
• turboderp for ExLlamaV3 and the EXL3 format, vcruz305 for the vllm-exl3 plugin, and coolbho3k for the calibration method the pack follows.
• vLLM for the engine and its renderer and tokenizer path.
• Unsloth for the GGUF our llama.cpp setup used.

AI assistance

AI agents (Claude and GPT models) designed and ran these tests. An independent AI reviewer checked each design before launch and audited each result afterwards, and an AI agent drafted this post. Artus set the goal, authorized the hardware and asked for these progress notes.