X local-model bench digest — Sep 25, 2026
As of Sep 25, 2026 — 12 posts collected from X. 31 measured runs catalogued, 12 kept to the digest only. Latest source post Sep 25, 2026 10:19 UTC.
Every number below is what its author published, transcribed with the engine, quantization and context length they reported. Sources are linked on each entry. Numbers without a full methodology stay out of the benchmark board until they are reviewed.
The 12 most recent posts are listed below; 11 older entries from this window are in the daily collection report (a digest-only entry never reaches the database).
DGX Spark × MiniCPM5-2B
Sep 25, 2026 · by @0xBakeer
| Context | Quant | Engine | Decode | Prefill |
|---|---|---|---|---|
| — | — | — | 100.8 tok/s | — |
| — | — | — | 89.7 tok/s | — |
| — | — | — | 69.5 tok/s | — |
| — | — | — | 55.5 tok/s | — |
| — | EXL3 | — | 47.9 tok/s | — |
| — | — | — | 39.2 tok/s | — |
| — | EXL3 + DFlash2 | — | 33.4 tok/s | — |
| — | 2-bit | — | 30.7 tok/s | — |
Everyone is posting 3 and 4 Spark clusters this week. Here is what ONE DGX Spark does for one user, all measured on my box: MiniCPM5-2B, OpenBMB's drafter on: 100.8 tok/s Qwen3.6-35B-A3B: 89.7 Ling-3.0-flash: 69.5 Qwen3.8-Flash-Next: 55.5 DeepSeek-V4-Flash: 47.9 in EXL3 Gemma-4-E2B: 39.2 Qwen3.8-27B: 33.4 EXL3 + DFlash2 Tinfield-1 at 2-bit: 30.7 One model at a time, 256 tokens in, 256 out
Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, hardware state).
8x 国产PCIe卡 × DeepSeek-V4.1-Flash
Sep 25, 2026 · by @NFT_Chen
| Context | Quant | Engine | Decode | Prefill |
|---|---|---|---|---|
| 1M | — | meta-infer | — | 13274.0 tok/s |
| — | — | community baseline | — | 1932.0 tok/s |
💥炸了！8张中国国产PCIe卡把DeepSeek-V4.1-Flash输入吞吐提升近7倍！ 不改权重、不改结构。是石科技 Meta-Infer 把社区 Day0 基线从 1932 tok/s 拉到 13274 tok/s，6.87倍，并支持 1M 超长上下文。 要点如下： 🔹PCIe卡没有高速互联，默认部署会退回低效实现，硬件被软件卡死 🔹先补齐内核：sparse-MLA Prefill + 替换慢GEMM，吞吐先到 5850 tok/s 🔹再重构通信与并行：算子融合、计算通信重叠、Prefill/Decode分策、缓存复用 🔹同套方法：V4-Flash +1.55倍；GLM-5.3 +1.92倍，首Token P95 141.6s→46.6s，上下文 27万→105万 ▫️MiniMax H3 视频生成端到端 +2.48倍 ▫️对比B300仍有差距（约3.5–5.6倍），但PCIe卡已从“能跑”变成“能打”…
Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, quantization, hardware state, measurement method).
M5 Ultra × Qwen 3.8 Flash Next
Sep 25, 2026 · by @Alisvolatprop12
| Context | Quant | Engine | Decode | Prefill |
|---|---|---|---|---|
| — | — | — | 85.9 tok/s | — |
| — | — | — | — | 6085.0 tok/s |
<Mac Studio M5 Ultra에 일시적으로 접근할 수 있게 됐습니다. 거의 최적화하지 않은 상태에서 Qwen 3.8 Flash Next가 긴 프롬프트에서 85.9 tok/s 속도로 실행되고, 6085 tok/s 프리필을 처리합니다.> (속도) 미친 거 아닌가요? ㅋㅋㅋ
Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, quantization, context length, hardware state, measurement method).
Mac Studio M5 Ultra × Qwen 3.8 Flash Next
Sep 25, 2026 · by @plotarmordev
| Context | Quant | Engine | Decode | Prefill |
|---|---|---|---|---|
| — | — | — | 85.9 tok/s | 6085.0 tok/s |
| — | — | — | 52.1 tok/s | 2960.0 tok/s |
Got temporary access to a Mac Studio M5 Ultra. Barely optimized, Qwen 3.8 Flash Next runs at 85.9 tok/s and 6085 tok/s prefill on long prompts. A 2x DGX Spark recipe gets 52.1 tok/s single stream and about 2960 tok/s prefill on 16k–64k prompts. Already ahead of 2x Spark, but I was expecting a much much better baseline…
Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, quantization, context length, hardware state, measurement method).
M2 Max × Qwen3.5-4B
Sep 25, 2026 · by @TeksEdge
| Context | Quant | Engine | Decode | Prefill |
|---|---|---|---|---|
| — | Q4 | transformers | 70.4 tok/s | — |
| — | Q4 | llama.cpp | 71.8 tok/s | — |
| — | Q4 | transformers | 15.9 tok/s | — |
| — | Q4 | llama.cpp | 13.4 tok/s | — |
| — | IQ4 | transformers | 60.2 tok/s | — |
| — | IQ4 | llama.cpp | 61.3 tok/s | — |
Hugging Face did Apple silicon a solid and users can use GGUFs directly. 🍎 🤯 Transformers can now load GGUF quants directly on Apple Silicon and keep the weights packed instead of expanding them back to BF16. It even reuses ggml Metal kernels underneath. On an M2 Max 🧠 Qwen3.5-4B Q4 Transformers — 70.4 tok/s llama.cpp — 71.8 🧠 Qwen3.8-27B Q4 Transformers — 15.9 tok/s llama.cpp — 13.4 🧠 Qwen3.5-35B-A3B IQ4…
Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, context length, hardware state, measurement method).
MI355X x8 × GLM-5.3
Sep 25, 2026 · by @vllm_project
| Context | Quant | Engine | Decode | Prefill |
|---|---|---|---|---|
| — | — | vllm | 469.0 tok/s | — |
Great work from the @TileRT_AI and @AIatAMD teams, who got GLM-5.3 to 469 tok/s single-user decode with vLLM on 8× MI355X on @SemiAnalysis_ AgentX. The run uses a disaggregated setup where vLLM handles prefill and TileRT handles latency-critical decode through vLLM's V1 connector interface. 1/2
Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, quantization, context length, hardware state, measurement method).
RTX 3060 × Bonsai 2, Qwen3.8-27B
Sep 25, 2026 · by @Oluwaphilemon1
| Context | Quant | Engine | Decode | Prefill |
|---|---|---|---|---|
| 125K | — | hermes agent | 22.0 tok/s | — |
| 125K | — | hermes agent | 26.0 tok/s | — |
This is what 12GB of VRAM can build in 2026. An RTX 3060 with 12GB of VRAM just spent 5 hours building an entire game locally. The setup: GPU: RTX 3060 12GB Model: Bonsai 2, Qwen3.8-27B Weights: 5.95GB Agent: Hermes Agent Context: 125K Speed: ~50 tok/s fresh, ~22 tok/s average Over the session, Bonsai 2 generated: • 328K tokens • 8 JavaScript files • 2,368 lines of code • An entire playable game • Zero handwritten…
Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, hardware state, measurement method).
RTX 3060 × qwen
Sep 25, 2026 · by @sudoingX · third-party report
| Context | Quant | Engine | Decode | Prefill |
|---|---|---|---|---|
| — | — | — | 26.0 tok/s | — |
| — | — | — | 50.0 tok/s | — |
holy shit! my bonsai2 mtp gguf now has 10,756 downloads in under 4 days, 6,780 of them since last night, for a model i packaged for gaming gpus like rtx 3060, 3070, 3080s and 40s. i am still processing this. if you're new here, this is bonsai 2 27b, qwen 3.8 27b dense compressed to ternary by prismml. it shipped without its mtp head, so i grafted the head back on and shipped a faster kernel for the gaming cards…
Digest-only: confidence 0.35 below 0.60; incomplete methodology (engine+version, quantization, context length, hardware state, measurement method).
DGX Spark x2 × DeepSeek-V4.1-Flash
Sep 24, 2026 · by @fkysly
| Context | Quant | Engine | Decode | Prefill |
|---|---|---|---|---|
| — | — | — | 54.5 tok/s | — |
现在社区最新已经能在2台 dgx spark 上，跑 deepseek v4.1 flash 解码达到 54.5 tok/s 了。不过看了一下。也是量化版本 https://t.co/vJ6ctXCOVO
Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, context length, hardware state, measurement method).
RTX 5070 Ti × MiMo-V2.6-Distill-Qwen-9B-Ablitrated
Sep 24, 2026 · by @Hikari_07_jp
| Context | Quant | Engine | Decode | Prefill |
|---|---|---|---|---|
| — | NVFP4 W4A8 | — | 108.0 tok/s | — |
Added quantization options to Hikari07jp/MiMo-V2.6-Distill-Qwen-9B-Ablitrated. NVFP4 W4A8 runs at 108 tok/s on an RTX 5070 Ti without speculative decoding. - Q4_K_M 5.63GB - NVFP4 W4A8 6.07GB - NVFP4 W4A4 7.11GB Worth a look if you have 12GB VRAM or less. https://huggingface.co/Hikari07jp/MiMo-V2.6-Distill-Qwen-9B-Ablitrated
Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, context length, hardware state, measurement method).
AMD Instinct MI355X x8 × GLM-5.3
Sep 24, 2026 · by @AIatAMD
| Context | Quant | Engine | Decode | Prefill |
|---|---|---|---|---|
| — | — | tilert | 469.0 tok/s | — |
How fast can an AI agent work as its context grows? @TileRT_AI reached 469 tokens per second with GLM-5.3 on 8 AMD Instinct™ MI355X GPUs Explore how TileRT and AMD are advancing inference for long-running agent workloads: https://www.amd.com/en/developer/resources/technical-articles/2026/tilert-agentx-on-amd-instinct-gpus.html
Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, quantization, context length, hardware state, measurement method).
M5 Max × Qwen3.8-Flash-Next
Sep 24, 2026 · by @jundotkim
| Context | Quant | Engine | Decode | Prefill |
|---|---|---|---|---|
| 16384 | — | mlx 0.7.0rc1 | — | 2007.0 tok/s |
| — | — | mlx 0.7.0rc1 | 131.5 tok/s | — |
| — | — | mlx 0.7.0rc1 | 136.9 tok/s | — |
oMLX 0.7.0rc1 is out! This release brings faster Qwen prefill & generation, MiMo V2.6, Ternary Bonsai 2, and partial block caching. DFlash now handles concurrent requests together, and Lightning MTP gets faster batch decoding! https://t.co/ozzrs4O5Sg Performance on M5 Max, 128 GB (Prefill, oQ4e quant) - Qwen3.8-Flash-Next: 1,522 - 2,007 tok/s (+32%) at 16K context. (Decode, batch=4, oQ4e quant) - Qwen3.8-27B with…
Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (hardware state, measurement method).
Verification status
This digest is an automated transcription of public posts. Runs with a stated engine build, quantization, context length, hardware state and measurement method are published straight to the benchmark board with a link to their source; the rest stay here. These are author-reported numbers, not an independent DiggTop re-measurement: every row carries its provenance so you can check the original, and numbers that fail review are removed. Our own testing follows the methodology. Spotted an error? Reply on the source post or submit a corrected run at /submit.
Sources
- @0xBakeer — Sep 25, 2026
- @NFT_Chen — Sep 25, 2026
- @Alisvolatprop12 — Sep 25, 2026
- @plotarmordev — Sep 25, 2026
- @TeksEdge — Sep 25, 2026
- @vllm_project — Sep 25, 2026
- @Oluwaphilemon1 — Sep 25, 2026
- @sudoingX — Sep 25, 2026
- @fkysly — Sep 24, 2026
- @Hikari_07_jp — Sep 24, 2026
- @AIatAMD — Sep 24, 2026
- @jundotkim — Sep 24, 2026
FAQ
Are the numbers in this digest verified?
They are transcribed from public X posts and the repositories those posts link to, with attribution and a source link on every entry. Rows that state the full methodology are published to the benchmark board automatically, so treat them as author-reported until a dispute is raised: the source link on every row is the check.
What does it take for a run to reach the verified board?
Engine and exact version, quantization, context length, hardware state and a described measurement method. Author-reported posts that state all five are queued as pending and can be promoted by a reviewer; anything missing a condition stays out of the database.
Where does the data come from?
Public X posts about local-model inference measured on owned hardware, discovered by keyword search every two hours and linked back to the original thread. The digest is regenerated daily.