X local-model bench digest — Oct 25, 2026

As of Oct 25, 2026 — 35 posts collected from X. 130 measured runs catalogued, 2 published to the board, 33 kept to the digest only. Latest source post Oct 25, 2026 09:12 UTC.

Every number below is what its author published, transcribed with the engine, quantization and context length they reported. Sources are linked on each entry. Only posts that state the full methodology (engine build, quantization, context length, hardware state, measurement method) reach the benchmark board; the rest stay here as leads.

GPU x3

Oct 25, 2026 · by @uzuki425

Context	Quant	Engine	Decode	Prefill
—	Q4_K_M	llama.cpp	32.5 tok/s	1472.0 tok/s
—	Q4_K_M	llama.cpp	52.7 tok/s	—

llama.cpp 高速化 3枚構成でVRAM稼ごうとしたら、Generateがおそい。たった32.5 tok/s、プロンプト処理（prefill）は1,472 tok/sもでてる。AIの判断「ドラフタを本体の最終層と同じ GPU（CUDA2）に置くと、MTP が効くようになり、Q4_K_M のまま生成が 32.5 → 52.7 tok/s（+62%）になりました。」

Digest-only: model.name missing; incomplete methodology (engine+version, context length, measurement method).

RTX 5090

Oct 25, 2026 · by @indesjyo · third-party report

Context	Quant	Engine	Decode	Prefill
—	4bit (self-developed)	yasakani	4383.0 tok/s	—

單張 RTX 5090 64 並行直接衝到 4,383 tok/s YASAKANI 是日本 22 歲開發者獨立寫的本地推理引擎在相同硬體、相同模型下單次生成比 vLLM 快約 1.6 倍 32 並行時比 llama.cpp 快超過 3 倍 Windows 原生、自研 4bit 佔用約 20.5GB 目前已準備開源 https://t.co/jrc9lulTdK

Digest-only: model.name missing; incomplete methodology (engine+version, context length, hardware state, measurement method).

RTX 3090 x2 × DeepSeek-V4.1-Flash-1017

Oct 25, 2026 · by @lxe

Context	Quant	Engine	Decode	Prefill
—	—	dsx	41.0 tok/s	—
3K	—	dsx	—	530.0 tok/s

Cleared 40 tokens/s. DeepSeek-V4.1-Flash running locally on 2× RTX 3090 + 128 GB RAM, using dsx and T16. 41 tok/s measured in free generation with expert deferral + FP8 projections. Rust, CUDA, and a lot of profiling. https://github.com/lxe/dsx

Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version).

RTX 5060 Laptop × Qwen3 8B

Oct 25, 2026 · by @pawankalyandev

Context	Quant	Engine	Decode	Prefill
—	Q4_K_M	llama.cpp	—	—
—	Q4_K_M	llama.cpp	—	—
—	Q4_K_M	llama.cpp	58.4 tok/s	—
—	Q4_K_M	llama.cpp	—	—
—	Q4_K_M	llama.cpp	—	—
—	Q4_K_M	llama.cpp	80.0 tok/s	—

so im building hiraishin been trying to push an 8b model past 100 tok/s on my laptop using speculative decoding, and somehow ended up spending most of my time figuring out why it was giving me different answers lol. im running qwen3 8b q4_k_m on an rtx 5060 laptop with 8gb vram, with qwen3 0.6b as the draft model. the whole point of greedy speculative decoding is that you get the exact same output as normal…

Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, context length).

RTX 5090 × Qwen3.8-27B

Oct 25, 2026 · by @niklaslenz_ai

Context	Quant	Engine	Decode	Prefill
262144	4-bit (MLX)	tensorfold 1.1.2 branch q35-prompt-int8 commit 18d93451	268.2 tok/s	—
262144	4-bit (MLX)	tensorfold 1.1.2 branch q35-prompt-int8 commit 18d93451	238.7 tok/s	—
262144	4-bit (MLX)	tensorfold 1.1.2 branch q35-prompt-int8 commit 18d93451	205.5 tok/s	—
262144	4-bit (MLX)	tensorfold 1.1.2 branch q35-prompt-int8 commit 18d93451	204.6 tok/s	—
—	EXL3 4.0 bpw	—	154.5 tok/s	—
—	EXL3 4.0 bpw	—	144.2 tok/s	—
—	—	ninfer	169.0 tok/s	—
—	—	ninfer	150.1 tok/s	—

Qwen3.8-27B 4-bit on one RTX 5090 decodes code at 268.2 tok/s, with BobClawblaw's TensorFold recipe and DFlash2. I ran his recipe the way his README does: TensorFold 1.1.2 on his branch q35-prompt-int8 (commit 18d93451), the MLX 4-bit weights, z-lab's DFlash2 drafts, FP8 KV cache, context 262,144 tokens. Speed run with thinking off, sampling from the Qwen model card. Decode, code: 268.2 tok/s Decode, prose: 238.7…

Full methodology stated (engine, quantization, context, hardware state, method) — 1 added to the benchmark board (1 of 8 measured rows queued; the rest are published here only).

RTX 5090 × Flint 40B-A2.5B

Oct 25, 2026 · by @yasakani_dev

Context	Quant	Engine	Decode	Prefill
—	—	yasakani	432.2 tok/s	—
—	—	yasakani	2961.0 tok/s	—
—	—	yasakani	4383.0 tok/s	—
—	NVFP4	vllm 0.32.1rc1.dev353+g2d40c0609	266.5 tok/s	—
—	NVFP4	vllm 0.32.1rc1.dev353+g2d40c0609	2819.0 tok/s	—
—	NVFP4	vllm 0.32.1rc1.dev353+g2d40c0609	3782.0 tok/s	—
—	Q3_K_L	llama.cpp b11804-c7fc50941	954.0 tok/s	—
—	Q3_K_L	llama.cpp b11804-c7fc50941	2969.0 tok/s	—

RTX 5090 1枚で、4,383 tok/s！シングルでも432.2 tok/s、vLLM比約1.6倍。ローカルAIエンジン「YASAKANI」の実測です。 ※4,383 tok/sは64件同時の合計処理速度（入力処理込み）。 Flint 40B-A2.5B／同じPC・同じGPUで比較 YASAKANI / vLLM（tok/s）シングル：432.2 / 266.5 32件同時：2,961 / 2,819 64件同時：4,383 / 3,782 llama.cpp Q3_K_Lとの32件同時比較は、2,969 / 954 tok/s（3.11倍）。モデル容量は20.5 / 20.6GB。…

Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (context length).

Apple M5 Pro 64GB × DeepSeek-V4.1-Flash (1017) GSQ-RCO

Oct 25, 2026 · by @ArnavPrabhu

Context	Quant	Engine	Decode	Prefill
—	IQ2_XXS	llama.cpp ds41-metal-runtime branch ista-metal commit cc5b…	—	252.0 tok/s
—	IQ2_XXS	llama.cpp ds41-metal-runtime branch ista-metal commit cc5b…	—	215.0 tok/s
—	IQ2_XXS	llama.cpp ds41-metal-runtime branch ista-metal commit cc5b…	—	222.0 tok/s
—	IQ2_XXS	llama.cpp ds41-metal-runtime branch ista-metal commit cc5b…	—	210.0 tok/s
—	IQ2_XXS	llama.cpp ds41-metal-runtime branch ista-metal commit cc5b…	16.3 tok/s	—
—	IQ2_XXS	llama.cpp ds41-metal-runtime branch ista-metal commit cc5b…	14.9 tok/s	—
—	IQ2_XXS	llama.cpp ds41-metal-runtime branch ista-metal commit cc5b…	26.3 tok/s	—

a 551B model on a 64GB mac ~250GB of weights. 64GB of RAM. it runs anyway prefill 5.7 → 252 tok/s decode 3.3 → 16.3 tok/s 128K context the SSD is the new VRAM https://github.com/arnavprabhu/deepseek-v4-flash-mac https://t.co/WdEXPJxcau

Full methodology stated.

GB300 NVL72 × DeepSeek-V4.1-Flash-1017

Oct 25, 2026 · by @NFT_Chen · third-party report

Context	Quant	Engine	Decode	Prefill
—	—	vllm	826.0 tok/s	—

😱逆天！开源直接打平官方！vLLM 用 DEP32 把 DeepSeek 自研引擎 DeepInfer 按在地上摩擦！DeepSeek-V4.1-Flash-1017 解码配置彻底炸裂！在官方同款解码配置下，131K 上下文，vLLM 吞吐与 DeepInfer 几乎打平（batch 8-256 差距 ≤6%），而且是用更轻的 DEP32 硬刚对方的 EP64DP64。 GB300 NVL72 整机 + P/D + Wide-EP 实测直接拉到 954K tokens/s per GPU（P90 196 tok/s/user），比之前单节点暴涨 7.9×；单用户专用解码路径更是冲到 826 tok/s。核心亮点 🔹CED 架构：Prefill 只激活 8B，Decode 激活 16B，Agent 长上下文成本直接砍半 🔹KV 仅 890 字节/token，1M 上下文轻松扛 🔹Engram + DSpark 5-token…

Digest-only: confidence 0.55 below 0.60; incomplete methodology (engine+version, quantization, hardware state, measurement method).

RTX 4090 48GB × DeepSeek-V4.1-Flash-1017

Oct 25, 2026 · by @MinLiBuilds

Context	Quant	Engine	Decode	Prefill
—	IQ2_XXS (2.00 bpw)	strata	5.3 tok/s	—
—	IQ2_XXS (2.00 bpw)	strata	42.6 tok/s	—

🚨 找到一个让模型速度提升 8 倍的隐藏瓶颈，改两个参数就解决了。一张 RTX 4090 48GB 跑 DeepSeek，刚开始生成速度只有 5 tok/s。折腾了一上午，查看日志，问题在SSD上。只改两个参数，速度直接飙升 8 倍！今天测试的是 DeepSeek-V4.1-Flash-1017，ISTA GSQ-RCO IQ2_XXS（2.00 bpw），使用 Strata 官方引擎。默认配置下，体验拉胯：生成速度：5.35 tok/s 首 Token 等待：33.36 秒长提示词首 Token：102.69 秒查日志挖出一个坑。模型的专家缓存命中率明明有 89.4%，但每生成一个 Token，仍然需要从 SATA SSD 读取约 77.7 MB 专家数据。我这台机器有 256GB RAM，内存足够了。于是只改了两个参数： --ssd-cache 0 --mmap-experts off 把约…

Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, context length, measurement method).

Apple M5 Pro × Bonsai 2 27B

Oct 25, 2026 · by @Oluwaphilemon1 · third-party report

Context	Quant	Engine	Decode	Prefill
—	—	—	28.0 tok/s	—
—	—	—	120.0 tok/s	—

Prism ML says it has compressed Qwen3.8-27B into a 5.95GB file using Bonsai 2, potentially bringing a 27B-parameter model within reach of GPUs with just 8GB of VRAM. The reported reduction is striking: from approximately 54GB to 5.95GB, making the compressed file about 9.1 times smaller. The model is called Bonsai 2 27B, and the supplied figures report 4.84 million Hugging Face downloads over the past 30 days. The…

Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, context length, hardware state, measurement method).

RTX 5070 × DeepSeek-V4.1-Flash

Oct 25, 2026 · by @0x0SojalSec

Context	Quant	Engine	Decode	Prefill
—	—	strata	117.0 tok/s	—

You can now run a DeepSeek-V4.1-Flash locally 12GB GPU and Offloads consumer hardware. - GSQ-RCO quantizations with Strata inference engine. - RTX 5070 with 64GB/ 105-117 tok/s - Prefill is much faster than decode not bad - More RAM to faster decode - engine uses layer-major chunked reading, - for fun is a great.

Digest-only: confidence 0.55 below 0.60; incomplete methodology (engine+version, context length, measurement method).

RTX 5090 × DeepSeek-V4.1-Flash-1017-GSQ-RCO

Oct 25, 2026 · by @ai_hakase_

Context	Quant	Engine	Decode	Prefill
26950	—	llama.cpp	10.5 tok/s	285.0 tok/s

【新モデルDeepSeek-V4.1-FlashのGGUF版が登場！】 ISTA-DASLabから独自のスマート量子化「GSQ-RCO」を適用した「DeepSeek-V4.1-Flash-1017-GSQ-RCO」のGGUF版がHugging Faceに公開されました！約2.00bpw版のモデルサイズは約257GBに達するため、VRAM単体では収まらずシステムRAMやストレージからの読込に依存する構成となります。現時点でllama.cppのオリジナルアーキテクチャサポートは未マージ（PR #28696 / #30817）のため、実行にはStrataのフォーク環境（XolotlやMayaなど）が必要ですね。 RTX 5090と128GB RAMの環境での実測では、長コンテキスト（約26,950トークン）において生成速度が約10〜11 tok/s、プリフィルが約270〜300…

Digest-only: no run with engine+version+quant+context+decode tps; incomplete methodology (engine+version, measurement method).

Also in this window

The 12 most recent posts are shown in full above; 23 posts from the same window are listed here in short form, each with its source post.

RTX 5090 × Qwen3.8-27B — 711.9 tok/s decode (8 measurements) · @niklaslenz_ai
8GB GPU × Uncensored Qwen 27B — no throughput stated · @0x0SojalSec
DGX Spark x2 × GLM-5.3-Flash — 188.5 tok/s decode (8 measurements) · @wei_wang
Unspecified platform × GLM-5.3-Flash — 78.0 tok/s decode (1 measurement) · @Bizuayeu
GPU × DeepSeek-V4.1-Flash — 196.0 tok/s decode (1 measurement) · @vllm_project
DGX Spark x2 × DeepSeek-V4.1-Flash-1017 — 141.2 tok/s decode (6 measurements) · @niklaslenz_ai
Unspecified platform × DeepSeek — 28.6 tok/s decode (2 measurements) · @lyesbahi
DGX Spark x2 × DeepSeek-V4.1-Flash-1017 — 141.2 tok/s decode (8 measurements) · @niklaslenz_ai
DGX Spark × Qwen3.8-27B — 1281.0 tok/s prefill (4 measurements) · @QwenAIFans
DGX Spark × Qwen3.8-27B — 67.0 tok/s decode (3 measurements) · @ashxhart
Unspecified platform × DeepSeek V4.1 Flash — 70.0 tok/s decode (1 measurement) · @buildwithhassan
Radeon AI PRO R9700 x2 × Qwen3.8 Flash Next — 259.0 tok/s decode (2 measurements) · @Vpoile1
GB300 NVL72 × DeepSeek-V4.1-Flash — no throughput stated · @inferact
RTX 5090 × Nemotron 3.5 Bolt E4B — 547.0 tok/s decode (4 measurements) · @BennyDaBall_OG
M3 Ultra × MiniMax M3.1 Flash — 141.0 tok/s decode (8 measurements) · @jundotkim
DGX Spark x2 × GLM-5.3-Flash — 99.2 tok/s decode (5 measurements) · @niklaslenz_ai
DGX Spark x2 × GLM-5.3-Flash — 99.2 tok/s decode (8 measurements) · @niklaslenz_ai
RTX 4090 48GB × Qwen3.8-Flash-Next — 157.0 tok/s decode (4 measurements) · @MinLiBuilds
RTX 5090 × DeepSeek-V4.1-Flash-1017 — 22.9 tok/s decode (3 measurements) · @niklaslenz_ai
RTX 5090 × DeepSeek-V4.1-Flash-1017 — 22.9 tok/s decode (3 measurements) · @niklaslenz_ai
Apple M5 24GB × DeepSeek-V4.1-Flash — 1.4 tok/s decode (7 measurements) · @mfranz_on
RTX 5090 × DeepSeek V4.1 Flash 1017 — 8.0 tok/s decode (1 measurement) · @jun1228909
DGX Spark x4 × Ling 3.1 Flash — 98.6 tok/s decode (2 measurements) · @sudoingX

How to read this page

This digest is an automated transcription of public posts. Runs with a stated engine build, quantization, context length, hardware state and measurement method are published straight to the benchmark board with a link to their source; the rest stay here. These are author-reported numbers, not an independent DiggTop re-measurement: every row carries its provenance so you can check the original, and a number that turns out to be misreported is corrected or removed. Our own testing follows the methodology. Spotted an error? Reply on the source post or submit a corrected run at /submit.

Sources

FAQ

Are the numbers in this digest sourced?

They are transcribed from public X posts and the repositories those posts link to, with attribution and a source link on every entry. Rows that state the full methodology are published to the benchmark board automatically, so treat them as author-reported until a dispute is raised: the source link on every row is the check.

What does it take for a row to reach the sourced tier?

Engine and exact version, quantization, context length, hardware state and a described measurement method. Posts that state all five are published to the benchmark board as soon as the collector reads them, each row carrying its source link — nothing waits in a human review queue. Anything missing a condition stays in this digest.

Where does the data come from?

Public X posts about local-model inference measured on owned hardware, discovered by keyword search every two hours and linked back to the original thread. The digest is regenerated daily.