Four 2017 V100s, One Fork, and a 27B Model That Refused to Fit

I posted earlier about putting four V100s in a box and running Qwen 3.8 on them. This is the post about how it actually works under the hood: the models, the tensor-parallel topology, the fork, and what it measures. Numbers below are all from my box unless I say otherwise.

The box

• AMD Ryzen Threadripper 2990WX (32 cores / 64 threads) on an ASRock X399 Taichi
• 128 GB DDR4 (yes, it gets used — vLLM wants host RAM for the model loader and the CUDA graphs, and the box hosts other things too)
• 4x Tesla V100 32GB, all PCIe Gen3, no NVLink — these are the desktop-adjacent cards, not the SXM2 datacenter parts
• ~$2.7k for the four cards plus their fans and shrouds; about $1.8k more for the motherboard, RAM, case, and power supplies, for ~$4.5k all in

The PCIe topology matters more than the card count here. The threadripper’s lane groups land in separate NUMA islands, and the four V100s split two ways: cards 0 and 1 on one island, cards 2 and 3 on the other. Links crossing between the two groups are not direct peer links — PCIe classifies them as “SYS”, the slowest topology — and the cards don’t even agree on width: 0 and 2 negotiated x8, while 1 and 3 negotiated x16. There is no NVLink to hide behind, so every tensor-parallel collective is a PCIe transfer. That one fact explains most of the performance stories below.

Why a fork at all

Upstream vLLM dropped Volta a while ago: the wheels need compute capability 7.5 or higher, and the 8-bit weight formats (AWQ, Marlin, FP8, W8A8) all hard-require Turing or newer. TensorRT-LLM removed Volta in v0.15. SGLang’s answer to V100 users is “use LMDeploy.” By 2026 the entire serious inference stack had quietly moved on from 2017 silicon.

The exception is 1Cat-vLLM, a vLLM engineering fork that treats SM70 as a first-class optimization target — their tagline is literally “Make Volta Fast Again.” It ships SM70 attention kernels, E4M3 FP8 KV caches on Volta, NVFP4 W4A16 execution, and a speculative-decoding stack tuned for Qwen on V100. That is why a 2017 card can run a current-gen 27B model at usable speeds today, and why this post exists. (My working copy also carries a handful of private patches on a mirror of the repo, all listed below.)

The two models

I run two vLLM engines, not one. The big model does the agentic work; a small fast model does titles, triage, and single-tool hops where waiting for the 27B to wake up makes the product feel broken.

Coder: Qwen3.8-27B

The main engine serves Qwen3.8-27B (the hybrid architecture: full-attention interleaved with 48 linear-attention / gated-delta-net layers, plus a vision tower) as the plain BF16 HuggingFace checkpoint — but quantized online to 8-bit per-block FP8 at load time, with the fork’s packed SM70 kernel (--quantization fp8_per_block, VLLM_SM70_FP8_QPN8=1).

“Online” is doing a lot of work in that sentence, because no off-the-shelf 8-bit checkpoint for this model loads on a V100. I tested the routes:

• The official FP8 checkpoint dies in the load path — the SM70 FP8 kernels cover a fixed shape set and one 4,120-wide layer falls through to Marlin, which requires widths divisible by 64.
• The INT8-W8A16 checkpoint is rejected at load: “Min capability: 75. Current capability: 70.” Even if it loaded, V100 has no INT8 Tensor Cores, and measured it crawls (18.9 tok/s vs 46 single-session on the FP8 route).
• The GGUF Q8_0 works in principle but the fork’s GGUF loader has no path for this hybrid architecture.

So I patched four small things in the fork to make the online block-FP8 route work on SM70: the quantizer’s capability floor now accepts 70 when the packed kernel is present; the odd-width layer (4,120 rows at TP4) is zero-padded to the kernel’s 128-row block and sliced back after the matmul; one compiled block-cast runs eagerly on SM70 because the compiler backend refuses FP8 on Volta; and the per-block linear path repacks into the packed layout at load instead of taking a generic fallback. Result: the 27B’s weights take 7.34 GiB per card instead of 14.28, single-session decode went from 31.8 to 46.2 tok/s, and the engine’s concurrency at the full window rose from 5 to 9 sessions.

Spark: Qwen3.5-4B

The second engine serves Qwen3.5-4B (same family, 32 layers), also BF16-on-disk quantized to per-block FP8 the same way. It lives on just two cards as a TP2 pair. I considered a single-card llama.cpp worker for this one and measured it: TP2 halves the per-card weight bytes and shards the KV heads, so the pair needs about 4.5 GiB per card instead of the 13 GiB a single-card GGUF worker holds — a meaningful difference on 32GB cards that also have to host a 27B model.

The topology: two engines, one quad

The layout:

Two things about this that aren’t obvious:

The small model is co-tenant on the big model’s cards, on purpose. Spark’s two x8 cards are also two of the four cards the 27B is sharded across. The engines divide the VRAM with utilization gates that sum to under 100% (the coder claims 0.75 of the card, Spark 0.18, and each one verifies free memory at startup before allocating). The window is deliberately 240k rather than the full 262,144 — that saves a little over a GiB per card, and when the small model is sharing two of those cards, that headroom is what keeps a routine long request from OOM-ing the big engine. It died twice in exactly that way before I found out.

Benchmarks

All measured on this box, one request at a time unless noted, cold prompts. Both engines run FP8 per-block weights (QPN8); the KV cache is FP8 (E4M3) in 2,048-token blocks, as the layout table shows.

The PCIe wall, quantified

With the big model at TP4, a torch profile of one prefill put tensor-parallel all-reduce at ~80% of CUDA time: about 145 collectives per engine step, two per layer (attention output, MLP down), each carrying ~21 MB. One 2,048-token prefill chunk moves roughly three gigabytes across the interconnect. Decode steps carry the same 145 collectives, which is why every decode step is dominated by its all-reduce too.

The fix was a single environment variable, not code: NCCL_P2P_LEVEL=SYS. NCCL’s default link classification refused to treat the cross-island links as direct peer links even though PCIe peer access works, and the ring routed them slower than they needed to. On the same 21 MB payload the standalone all-reduce dropped from 13.3 ms to 5.67 ms, and live prefill throughput went from ~913 to ~1,583 tok/s.

The honest ceiling, from a TP sweep (same model, 8,209 in / 513 out, one request):

Half of every TP4 decode step is fabric overhead. Batching is what hides it: the cost is per step, not per token, so six concurrent 256-token requests ran at 188.7–190.1 tok/s aggregate (33.8 each) even though a single stream caps out near 50.

Prefill at long context: TP2 vs TP4

Same model, same flags, only the card count and pair change. Medians of three cold runs each, 2,048-token chunks:

Doubling the cards buys only 4% at 32k but 24% at 200k — this prefill is not FLOP-bound. The cost is per step, and it does not scale with rank count: the ~145 all-reduce transfers per 2,048-token chunk ride PCIe either way, and at TP2 each rank runs attention over a doubled KV shard. A chunk’s wall time creeps from about 1.2 s at 32k to 1.7 s at 200k on TP4, and the gap between the two configurations widens with the context for exactly that reason.

Decode and repetition

• Decode speed is flat with respect to context length on this model/architecture — no long-context collapse in practice.
• Prefix caching is real and it’s a cliff: a 13,295-token prompt took 7.93 s to first token on first call and 0.73 s on the repeat (12,288 tokens cached). New context is what costs; repeating your system prompt plus history is nearly free.
• Sustained mixed load with 7–8 sessions running: the engine does about 33 tok/s total, so roughly 4 tok/s per session. Under an eight-way simultaneous fresh-context prefill, per-session decode briefly drops to ~1.9 tok/s — the freeze is once per prefill chunk, not continuous starvation.
• Day to day, for what a single interactive session actually feels like: 20–40 tok/s. That’s fine for reading, slow for watching it think, and the fans compensate with volume.

The 8-bit route, before and after

What it costs

About $2.7k for the four cards, their fans, and the shrouds, plus roughly $1.8k for the platform — motherboard, RAM, case, and power supplies — for about $4.5k all in. vLLM wants more host RAM than you’d expect, and cold starts with an empty compile cache take over eight minutes (warm restarts about two and a half). There’s no NVLink, and if I had bought the SXM2 parts instead the TP4 decode would have roughly doubled — but those cards are five times the money and a chassis away from hobbyist.

Compare these numbers to the alternatives. The DGX Spark is $4,700 from Nvidia for the 4 TB model, and more on eBay; it runs Qwen3.8-27B at 11.8 tok/s single-stream (7.7 at 128k context, 4-bit llama.cpp) and around 33 with multi-token prediction, and its batched serving scales with concurrent sessions. The Mac Studio M5 Ultra starts at $5,499 with 96 GB — plenty for the 8-bit 27B — and 256 GB runs $9,499; it hits 46.3 tok/s single-stream on 4-bit, still 30.1 at 128k, and about half that at 8-bit since a dense model is bandwidth-bound. This box’s numbers are 8-bit weights with an FP8 E4M3 KV cache, and at six concurrent sessions it puts out ~190 tok/s aggregate on the 27B.

For what it’s actually doing for me: this box replaced ~$200/month of API spend with a machine that costs pennies in electricity, and the 27B handles real multi-hour agentic sessions at a concurrency I couldn’t get any other way at home. The fans are loud, the wiring is annoying, and half my time was spent teaching a fork to load an odd-shaped layer — but it works, and works well.

Editor’s note: this post was written by qwen3.8 27b based on memories kept in my memory service (a qdrant backed self maintained RAG).