Last updated: 7 November 2026, 16:28 UTC. This page changes as new measurements land, and the changelog at the end lists each change. Rows still being measured are marked as such, never estimated. The day-by-day notes are Parts 1 to 5.

We run DeepSeek-V4.1-Flash-1017, an open model with 763 billion parameters, on two RTX PRO 6000 Blackwell cards in a workstation at home. As released, it doesn’t fit on two cards. So our server is a Frankenstein: diffbot’s 3-bit version of the model and their serving recipe, parts from other engines, and parts of our own. This page answers four questions in one place:

• How fast is it, against the same model on other setups?
• How much faster has it become since our first version?
• What do other models do on the same kind of cards?
• Is the speed worth it?

At a glance

• Our 2-card server needs 34.8 s of model time per task on our agent test, or 39.0 s with tools included.
• NVIDIA’s full-quality version on 4 cards needs 36.4 s, with its drafter set the same way as ours. The two are about level. Ours takes about 4.5% less model time, which is less than the spread between single draws. So our 3-bit Frankenstein gets 4-card full-quality speed from half the cards. The price is quality that hasn’t been measured on public benchmarks yet.
• A second 4-card run, with vLLM’s default draft setting, needed 42.8 s. It accepted fewer of the drafter’s guesses and thought longer. Whether the setting explains the gap isn’t settled with 3 draws each.
• On AWS’s own 2 cards, diffbot’s recipe as published needs 58.9 s. That’s the same 3-bit pack, without our parts. The gap down to our 34.8 s is the home hardware plus our parts, and we haven’t split it further.
• Squeezing the full-quality model onto 2 cards costs about 9.5x: 330.5 s with its drafter (352.9 s without), because the CPU has to compute most of the model. A 2-bit version small enough to fit entirely on 2 cards needed 150.2 s on ExLlamaV3’s own server, because diffbot’s engine couldn’t load it; its drafter wasn’t tuned.
• Since our first version, we’re about 2x faster, from 82.6 s to 39.0 s per task. The first figures come from an older version of the test.
• Quality: our 3-bit version has no public benchmark score of its own. On our small test, it passed 83% of tasks. The other setups passed 78% to 94%, on too few tasks to rank them.

How we measure

• The test is our 6-task agent test:
  • two questions about a document of 128,000 or 200,000 tokens;
  • four coding tasks in a code base, with tools to search, read and write files and to run tests.

  Grading is mechanical: exact facts and line citations for the questions, hidden tests for the code. A draw is one pass over all 6 tasks.

• Model seconds per task are the time the model spends reading and writing, on the server’s own clock, averaged over the tasks. This is the number we rank by.
• Task seconds per task add the tools and the network, on the client’s clock.
• Single-user speed: every number on this page is for one request at a time. Serving several users at once is a separate question; see Throughput.
• Every DeepSeek row uses the same settings: temperature 1.0, top-p 0.95, thinking on and one request at a time.
• Drafters: a drafter is a small model that guesses a few tokens ahead, and the big model checks all the guesses in one step. It changes the speed, not what the model would write. Its guesses can be its single top choice (greedy) or sampled from its probabilities (probabilistic), and that can change how many guesses are accepted. Our server samples them. Every setup gets its drafter unless the row says otherwise. Short tests picked how many tokens ahead it guesses for NVIDIA’s FP4 and for full quality on 2 cards. Ours kept 5 after a test against 7 on 1 November. diffbot’s recipe as published and the 2-bit version used 5 without a test.
• Where the tools run: in a sandbox on our Mac at home. The AWS runs reach the Mac over the internet, so their task seconds include extra network time. That’s why we rank by model seconds.

Is it worth it?

Each point is one model on one setup. Across is its time per task. Up is the score its maker reports on DeepSWE v1.1, a public benchmark of agentic coding. Up and to the left is better. DeepSWE v1.1 is the one score that all three makers report. Arrows go from a Qwen release to our tuned stack for it.

• Time: DeepSeek’s points use model time. The Qwen points use task time, from October runs with greedy decoding. Those runs ran the tools on the GPU machine itself, so there was no internet hop, and they were graded with a variant that accepts one more wording on one question.
• Hollow points have no score of their own. Their weights are at lower precision than the release the score is for, so that score is an upper bound for them:
  • our 3-bit version, under DeepSeek’s 77.0 for the full model;
  • the Qwen models on our tuned stacks, which run lower-precision weights than Qwen’s releases;
  • the 2-bit version that fits on 2 cards. turboderp reports 90.4% on GPQA Diamond for it, against 90.2% for the original (a figure turboderp took from OpenRouter), but it has no DeepSWE score.

  NVIDIA’s FP4 version is drawn solid because NVIDIA reports its own scores, within 0.7 points of the original on six benchmarks. Every DeepSeek point on the chart has its drafter, and NVIDIA’s 4-card run is the one with the same draft setting as ours.

• Harnesses: the makers ran DeepSWE in different harnesses:
  • DeepSeek used Mini-SWE-V2;
  • Qwen3.8 27B used Claude Code;
  • Qwen3.8 Flash Next used the better of Claude Code and mini-SWE-agent.

The same model on other setups

Setup	Cards	Drafter	Model s per task	Task s per task	Passed	Writes, tokens/s	Draws
Ours: the 3-bit Frankenstein	2 Workstation, home	DSpark 5	34.8	39.0	270/324	259.4	54
NVIDIA’s FP4, full quality	4 Server, AWS	DSpark 5	36.4	59.3	15/18	244.8	3
NVIDIA’s FP4, vLLM’s default draft setting	4 Server, AWS	DSpark 5, greedy	42.8	63.9	15/18	227.0	3
NVIDIA’s FP4, no drafter	4 Server, AWS	none	76.1	103.8	14/18	102.5	3
diffbot’s recipe as published	2 Server, AWS	DSpark 5	58.9	82.9	17/18	172.6	3
2-bit, all on 2 cards	2 Server, AWS	MTP 5, not tuned	150.2	195.9	14/18	88.8	3
Full quality on 2 cards	2 Server, AWS	DSpark 3	330.5	364.3	10/12	26.5	2
Full quality on 2 cards, no drafter	2 Server, AWS	none	352.9	384.4	16/18	24.3	3

The setups:

• Ours: diffbot’s EXL3 3-bit pack in vLLM, plus our parts, on RTX PRO 6000 Workstation Edition cards at home.
• NVIDIA’s FP4: full quality where it fits. This is NVIDIA’s 4-bit version in vLLM, on 4 RTX PRO 6000 Server Edition cards.
  • The first row samples the drafter’s guesses, as our server does.
  • The second keeps vLLM’s default, the drafter’s top guess.
  • The row without a drafter is there only to show what the drafter is worth.
• diffbot’s recipe as published: the same 3-bit pack and serving recipe as ours, exactly as diffbot published them, on AWS’s 2 cards. Our parts aren’t in it. The CPU computes its rarest experts on 2 × 11 cores of a 24-core Xeon, against 64 Zen 5 cores at home.
• 2-bit, all on 2 cards: turboderp’s EXL3 at 2.25 bits per weight, small enough that every weight fits on the cards.
  • diffbot’s engine can’t load it: turboderp quantizes every layer, and diffbot’s engine expects that only for the experts.
  • So it ran on TabbyAPI with ExLlamaV3 1.6.6, using its MTP head as the drafter, 5 tokens ahead. We didn’t test other lengths, so its drafter isn’t tuned.
  • The engine and the quantization both differ from the other rows, so this row doesn’t isolate either one.
• Full quality on 2 cards: the original model on 2 cards, with most of its experts computed by the CPU (SGLang and kTransformers).
  • The first row has DeepSeek’s DSpark drafter, guessing 3 tokens ahead, which beat 5 in a short test.
  • It differs from the row without a drafter in two more ways. A one-line patch was needed before the drafter would load at all. And the recipe’s KV-memory lending is off: it keeps more experts on the cards in spare KV-cache memory, and with it on, the first drafted step crashed.
• DSpark 5 and DSpark 3 are DeepSeek’s DSpark drafter, guessing 5 or 3 tokens ahead.

Notes:

• Single-user speed: every row is one request at a time. How many users each setup serves at once is not measured yet (Throughput).
• Hardware differs. Ours runs on Workstation Edition cards at home, capped at 500 W while the CPU is busy and at 600 W otherwise. The AWS rows run on Server Edition cards, and the 2-card ones on a smaller CPU. Our parts aren’t on AWS, so diffbot’s recipe as published is the AWS baseline. On the same AWS cards, the 4-card FP4 run takes 0.62x its time, the 2-bit run 2.55x, and full quality on 2 cards with its drafter 5.6x.
• What the drafter is worth: on the 4-card setup, it cuts model time from 76.1 s to 36.4 s (2.1x). The two drafted runs differ by 6.4 s per task. The run with vLLM’s default accepted fewer guesses per step (2.45 against 2.78), and its draws thought 12% longer. That the draft setting causes the gap is a hypothesis with 3 draws each, not a result. The draft length of 5 tokens matches our server; a short test of 3, 5 and 7 before the first drafted run found 3 and 5 tied.
• With the experts on the CPU, the drafter is worth little: full quality on 2 cards took 330.5 s with it, against 352.9 s without (0.94x). Read that as no worse, maybe 5 to 10% better, not as a measured gain: the draws without it ranged from 306 to 377 s per task, and the two setups differ in more than the drafter (above). The CPU still computes most of the experts, and a drafter can’t change that. Our reading, not measured: each guess the big model checks costs CPU work too.
• The 2-bit version writes slower as the context grows: 77 tokens/s at 128,000 tokens in a short test, against 163 to 184 at 8,000 to 64,000.
• Few draws on AWS: 3 draws each (18 tasks), and 2 for full quality on 2 cards with its drafter (12 tasks), against our 54. Pass counts from 12 or 18 tasks are too few to rank quality. Every failed task on AWS was one of the two document questions, on citations, and every coding task passed.
• The “original”: DeepSeek ships the model’s experts in 4-bit form, so the original is already 4-bit where most of the weights are. NVIDIA’s FP4 version makes their arithmetic 4-bit too, and NVIDIA reports it within 0.7 points of the original on the six benchmarks it ran.
• Our row was measured on 5 November. The part we added since (our top-k kernel, Part 5) saves about 0.3 ms per decoding step, too little to show up in task time with the rows run so far.
• Writing speed is on the server’s clock, while the model writes.
• Mac load: five AWS draws ran while our Mac had extra work on it: two of the 4-card draws without the drafter, one of diffbot’s recipe as published and two of the 2-bit version. That affects only their task seconds; model time is counted on the AWS machine.

How much faster than our first version

Version	Date	Task s per task	Draws	Test version
The first version we served: llama.cpp with Unsloth’s 2-bit GGUF and the DSpark drafter	31 Oct	82.6	3	older
diffbot’s recipe (vLLM and their 3-bit pack) as they published it	31 Oct	59.3	3	older
The same, on today’s test	1 Nov	46.7	3	today’s
Today’s served Frankenstein	5 Nov	39.0 (model time 34.8)	54	today’s

• About 2x faster than the first version we served (82.6 s against 39.0 s per task).
• About 1.2x faster than diffbot’s recipe as published, when both ran today’s test (46.7 s against 39.0 s). The 31 October run of that recipe took 59.3 s on the older test.
• Not one clean series: the two test versions differ in the facts of one question, and the early rows are only 3 draws each. A clean rerun of the published recipe on today’s test is queued on our box.
• Since 2 November, task time has stayed flat inside its spread. The gains since then are inside each decoding step. A step took 14.42 ms on 5 November’s rows, and our own top-k kernel took about 0.31 ms off that (Parts 4 and 5).

Other models on the same kind of cards

From our October campaign with Qwen3.8 models, on AWS’s Server Edition cards.

Model	Cards	Drafter	Task s per task: as released, then on our tuned stack	Draws	DeepSWE v1.1
Qwen3.8 Flash Next	2 Server, AWS	MTP, 3 tokens	77.7, then 49.9	1, then 3	58.7
Qwen3.8 27B	1 Server, AWS	MTP, 5 tokens	181.3, then 140 to 145	1, then 3 (two stacks)	42.2
For scale: DeepSeek-V4.1-Flash-1017, ours	2 Workstation, at home	DSpark, 5 tokens	39.0 (model time 34.8)	54	77.0 for the full model

• Different settings from the DeepSeek rows: these runs used greedy decoding, ran the tools on the GPU machine (no internet hop) and were graded with the variant described above. Their times are task time.
• Our tuned stacks run the models at lower precision than the releases, with our own kernels, among them SilentMountain’s decoding kernel for Flash Next’s experts. In October we checked them against the releases on our own tests (math questions, needles in long documents and this agent test), but they have no public scores of their own.
• A rerun of Flash Next on our home box with the DeepSeek rows’ settings is queued. It replaces this row when it lands.

Throughput: several users at once (measuring)

Everything above is single-user speed. How much a box can serve to many users at once is a different question, and we have no concurrency numbers yet for either model. This section will show none until we measure them. What we know so far:

• Qwen3.8 Flash Next fits on one card. NVIDIA’s 4-bit version of it, on our tuned one-card stack, took 55.5 s per task (one draw, 5 of 6 tasks passed; AWS, greedy decoding), against 49.9 s on two cards together. So a 2-card box can run two copies at once, one per card.
• Each copy should also serve many users at once cheaply. It uses only about 6 billion of its parameters per token, and its linear-attention layers (Gated DeltaNet) keep a fixed-size state per request. This is an expectation, not a measurement.
• Our DeepSeek setup needs both cards for one copy, and the CPU computes its rarest experts, which likely limits how many users it can serve at once. This isn’t measured either.

Still measuring

• diffbot’s recipe as published, rerun on today’s test on our box.
• Qwen3.8 Flash Next on our box, with the DeepSeek rows’ settings.
• Throughput with several users at once, for both models.

The story so far

• Part 1: the body, vLLM with diffbot’s 3-bit pack in place of llama.cpp, and a faster schedule for the experts the CPU computes.
• Part 2: a part from another fork, vLLM-Moet’s overlap for these cards, and a cache for repeated prompts.
• Part 3: a plan for which experts the CPU computes, made from our own agent traffic, and a newer vLLM base we turned down.
• Part 4: our first own part, an exact top-k kernel for the attention indexer.
• Part 5: that kernel in served traffic, 0.31 ms faster per decoding step.

Credits

• DeepSeek for DeepSeek-V4.1-Flash-1017 and its DSpark drafter.
• NVIDIA for the FP4 version.
• vLLM for the engine of the 4-card runs and of our server.
• diffbot for the EXL3 3-bit pack and the serving recipe our server builds on, also run as published on AWS.
• turboderp for EXL3, ExLlamaV3 and the 2.25-bit version.
• theroyallab for TabbyAPI, which served the 2.25-bit version.
• SGLang and kTransformers (kt-kernel) for the run with the experts on the CPU.
• Unsloth for the GGUF of our first version, and llama.cpp for its engine.
• The Qwen team for Qwen3.8 Flash Next and Qwen3.8 27B.
• kacper-daftcode for vLLM-Moet’s SM120 overlap, still served.

AI assistance

AI agents (Claude and GPT models) designed, ran and checked these measurements and built our parts of the server, and an AI agent drafted this page. Artus set the goal, approved the hardware and the AWS budget, and asked for this overview.

Changelog

• 7 November 2026, 16:28 UTC: one wording fixed: our server’s draft length was tested too (5 kept over 7, on 1 November), so the drafter note now says so.
• 7 November 2026, 16:20 UTC: the first table’s last measuring row is in:
  • Full quality on 2 cards with its drafter took 330.5 s (2 draws), against 352.9 s without. That’s no worse, maybe 5 to 10% better, inside the spread. So squeezing the model onto 2 cards costs about 9.5x ours, not 10x.
  • On the chart, the full-quality point moved to 330.5 s, so every DeepSeek point there has its drafter.
  • The 2-bit row now says its drafter wasn’t tuned.
  • The Mac-load note now names all five AWS draws that ran with extra work on our Mac.
• 7 November 2026, 13:33 UTC: AWS round 2 changes the headline:
  • About level now, not 1.2x. NVIDIA’s 4-card FP4 run with the same draft setting as ours took 36.4 s, against our 34.8 s, so “about 1.2x faster” no longer stands. The 42.8 s run stays, as a second run with vLLM’s default draft setting.
  • The “ours on AWS’s cards” row is replaced by diffbot’s recipe as published on AWS (58.9 s), because our parts aren’t on AWS.
  • The 2-bit version that fits on 2 cards took 150.2 s, on TabbyAPI and ExLlamaV3.
  • New on the chart: the 4-card point moved to 36.4 s, and the 2-bit point is added.
• 7 November 2026, 12:50 UTC: the first table made narrower, with the setups described below it; no numbers changed.
• 7 November 2026, 12:48 UTC: a note that every number is single-user speed, under the first table and on both charts, and a Throughput section, still measuring.
• 7 November 2026, 12:43 UTC: first version. It has 4 setups of DeepSeek-V4.1-Flash-1017 on our test (3 of them new, run on AWS on 7 November), our progress since 31 October, two Qwen3.8 models from October, and the time-against-score chart. Five rows are still measuring.