256 experts, 512 lines: I stuffed a 744B model into an engine written for a different one

DwarfStar (ds4.c) isn’t mine: it’s a native C inference engine, MIT-licensed, written to run DeepSeek V4 — Flash and PRO — on high-end personal machines. Metal, CUDA, ROCm backends; KV cache streamed from SSD; distributed inference. All in a single tree of .c files with no external runtime dependencies. It’s beautiful and deliberately narrow: one model at a time, validated against the official logits. The question I asked myself was simple: what if I stuffed a different model in? Not DeepSeek. GLM-5.2, ~744 billion parameters, Zhipu’s GlmMoeDsaForCausalLM family.

The short answer: you can. The long answer is that a “narrow” engine hides its narrowness in innocent-looking constants. And the first one that bites you is the number 256.

Why GLM-5.2 in an engine built for DeepSeek

It’s not masochism. GLM-5.2 is architecturally a cousin of DeepSeek-V3.2: MLA + DSA indexer + MoE with shared expert + MTP. It’s exactly the path DS4 already implements. The differences are deltas, not chasms: no multi-Head-Compression (n_hc=1), no V4 KV compressor (dense attention), sigmoid routing with top-8 instead of top-6, rope θ=1e6, and an “absorbed” MLA with head_dim = kv_lora + rope = 512 + 64 = 576. A standalone converter remaps the GlmMoeDsa safetensors into a GGUF with DS4 layout, and I verified that the absorbed attention matches GLM’s explicit one down to 2e-08. On a small model with the real architecture, DS4 reproduces HF transformers logits at corr 0.9994 / max|Δ| 0.035 (quantization noise). The native GLM tokenizer (154880 tokens / 321649 merges) lives inside the GGUF and DS4’s byte-level BPE reproduces it exactly on 6 out of 6 different strings.

The real test is the real model: q2 GGUF at ~252 GB, resident on 2× H200 NVL (143 GB each, no NVLink), CUDA 13.2. Prefill ~34 t/s, generation ~16 t/s. -p "What is the capital of France?" → “The capital of France is Paris.” It works.

The constant that always said 256

But before I got there, the MoE router on CUDA slammed the door in my face. DeepSeek Flash has 256 experts; GLM has a different number, with top-8 and sigmoid scoring instead of top-6 scale-1.5. The router kernel wasn’t configured for 256: it was hardwired to 256. Logit strides, top-k bounds, the 1.5 scale — all literal in the body of the kernel:

// before: DeepSeek's shape carved into the kernel
const float *log = logits + (uint64_t)t * 256;
for (int i = 0; i < 256; i++) prob[i] = sqrtf(softplus_dev(log[i]));

// after: parameterized, with the 256 fast-path intact
const float *log = logits + (uint64_t)t * n_expert;
for (uint32_t i = 0; i < n_expert; i++) prob[i] = sqrtf(softplus_dev(log[i]));

The rule I set for myself — and it holds for all of this work — is: zero regressions on the DeepSeek path. The n_expert == 256 case stays on the existing warp/parallel kernels, byte-identical; only the different counts get routed to the parameterized serial kernel. I verified that Flash runs at 40.8 t/s with --logprob-vectors on the same build. This router generalization landed in an upstream PR (#466).

A little further along, the same story with the DSA indexer: top_k hardwired to 512 (Flash’s value). PRO uses 1024, and on --cuda it emitted control-character garbage on every prompt, because if (top_k > 512u) return 0; no-opped the entire indexed attention, and the shared-memory arrays (comp_rows[512], scores[768]) truncated the 1024 selected rows. Widened to [1024]/[1280] and raised the dispatch wall: from garbage to “The capital of France is Paris.”, with think mode reasoning and then answering. Bonus: the parallel top-k instead of the single-thread fallback gave +27% (gen 1.69 vs 1.33 t/s). Flash regression byte-stable.

The thorn: handshake complete, then dead with 0.5 GiB free

The moment that cost me half a day wasn’t a wrong kernel. It was a memory heuristic that made sense on one machine and not on the other.

GLM-q2 loaded. The two H200s completed the distributed handshake. And then — at session-creation time, after the weights were already in and the handshake had succeeded — OOM. The prefill graph couldn’t find memory. With 0.5 GiB free on a 143 GB card.

The culprit was cuda_q8_f16_cache_reserve_bytes(). The q8→fp16 dequantization cache is eager: it fills HBM up to the reserve. Under 112 GiB the reserve is “5% or 4 GiB, whichever is larger”. But on cards ≥ 112 GiB there was a special case that returned a flat 512 MiB:

// the special case that starved the graph on big cards
if (total_bytes >= 112ull * 1024ull * 1024ull * 1024ull) {
    return 512ull * 1048576ull;   // 512 MiB, period.
}

On an H200 the optional cache ate 14 GiB and left half a gig for the session graph, the cuBLAS workspaces, the transient buffers. And DS4_CUDA_WEIGHT_CACHE_LIMIT_GB doesn’t touch this cache. Worse: loading an MTP model disables it, so the bug hides precisely in the tests that look innocent. The patch is to remove the special case — 5%/4 GiB for everyone. Before: OOM (14 GiB cache, 0.5 free). After: 7 GiB free, runs at 16 t/s. It’s the CUDA twin of a bug already known on the ROCm runtime (PR #446); I sent it up as PR #472.

The diary lesson: an eager reserve that scales badly is invisible until you change the card’s order of magnitude. The machine the code was born on didn’t have 143 GB cards. Mine did. The bug was there all along, waiting for a model big enough.

Where my contribution ends (diary honesty)

• GLM-5.2 backend: it works, it generates coherent text, resident on 2× H200. I posted it as a report, not a mergeable PR: the diff on the engine is large and experimental, all gated on n_hc==1 so Flash/PRO stay byte-identical. The Python converter is standalone, not in the tree.
• CUDA fixes (router 256→generic, indexer 512→1024, cache reserve): these yes, model-independent, upstreamed as separate PRs (#466, #472). The router for now accepts 256 or 384, it’s not arbitrary yet.
• Serving: ds4-server that speaks GLM-5.2 on /v1 (chat template + multi-EOS stop), a /health route that becomes ready only when the distributed route is complete, plus the gateway wrapper. Plumbing stuff, but it’s what makes the model usable and not just runnable.
• DSpark (DeepSeek’s block-based speculative decoding, DFlash + markovian head): here I’m being honest — there’s the complete design and the loader + --dspark flag that loads and validates the GLM drafter (5 Qwen3-style layers, 7.1 GB), verified on the box. But the draft runtime — the forward graph, the hidden capture, the block-K verification — is NOT built yet. It’s scaffolding. I’m not pretending it generates.

Why bet on it anyway? Because the numbers, not the theory, justify it. I measured DSpark on llama.cpp (Qwen3-8B, same H200 box): 0.89× on a fast target, 1.74× on a slow target, with a clean monotone curve — break-even around 130 t/s. GLM-q2 runs at 16 t/s (62 ms/token): deep in the favorable regime. The expected speedup is ≥1.7×. Speculative decoding is regime-dependent, and it’s the same lesson I learned with PRO’s MTP: on cheap tokens the draft+verify overhead eats the gain; on expensive tokens it pays.

What I learned

That a “narrow and fast” engine pays for its speed in constants. 256, 512, 1.5, 112 GiB — these aren’t random magic numbers, they’re the shape of a specific model carved into the hot code. Porting a different model onto it isn’t writing an adapter: it’s hunting down every point where someone wrote the dimension they had on hand instead of the one the model declares. And the trickiest part isn’t the wrong kernels — those scream, they spit control characters. It’s the reasonable heuristic that starves the graph with half a gig free, after a perfect handshake, just because your card is bigger than the author’s. Generalization, when you do it right, isn’t “make everything configurable”: it’s isolating the existing fast-path, leaving it byte-identical, and routing only the new case. No regressions for whoever came before you.

DwarfStar / ds4.c is a third-party MIT project (© The ds4.c authors). Here I’ve told the story of my exploration: an experimental GLM-5.2 backend and a few model-independent CUDA fixes proposed upstream. The GLM-5.2 code and the converter stay out of the tree, for now.