Inkling on Two DGX Sparks: The Only vLLM Lane, and the Two Walls Behind It

Matt Suiche · · 985 words · 5 minute read

Thinking Machines shipped Inkling-Small at the end of July: a ~300B-total, ~10B-active hybrid MoE (short-conv plus relative-bias attention), natively multimodal, 1M context, and - the part that matters for homelab hardware - released as NVFP4 from day one. 170.7 GB of weights. Two DGX Sparks hold 243 GB of unified memory. You can see where this is going.

For five weeks, though, every Spark recipe for it - drowzeys’ champion image, MiaAI’s wrapper, and the half-dozen forks - ran on SGLang with a custom-baked image, for a simple reason: SGLang had Inkling support in early August, and vLLM only gained it this week, in v0.28.0. The engine we build everything on could not even load the model before Monday.

Our lane runs on stock vllm/vllm-openai:v0.28.0 plus two hotfix files applied at container start. CUDA graphs on. No eager mode. No forked runtime. It is, as far as I can tell, the only vLLM deployment of this model on this hardware anywhere. This post is about the two walls that stood between the day-0 image and a working server, because both of them will outlive Inkling: they are GB10 lessons, not Inkling lessons.

Wall 1: sm_121 cannot run this attention shape 🔗

Inkling’s attention is hard-wired to FA4 relative-bias kernels. On GB10 (Grace-Blackwell, compute capability 12.1), both FA4 backends in vLLM v0.28.0 fail, for two different reasons:

  1. The gate mis-selects. _use_sheared_bias() checks capability.major in (10, 11) - an enumeration that predates sm_12x - so GB10 gets routed to the Hopper cute score-mod path, which asserts Paged KV not supported on SM 12.0. Guaranteed failure, and it fires inside the JIT warmup loop, which is why the engine used to die ~28 seconds after the last weight shard with no traceback.
  2. The intended path is shape-incompatible. Patch the gate and tml_fa4 gets much further - KV sizing completes - then dies at assert tile_n == 128. Inkling is a diff-headdim model (head_dim 128, v_head_dim 64, rel_extent 1024); the SplitKV heuristic shrinks tile_n to 64 for smem reasons, and the rel_bias metadata hard-requires 128x128 tiles.

Fifteen configuration knobs were eliminated before the probes isolated this. The community reached the same conclusion from the SGLang side: their champion recipe also abandons FA4 and runs a Triton attention lane.

Kernel dispatch on GB10 for Inkling’s rel-bias attention: the stock gate mis-routes sm_121 to the Hopper cute path (assert: no paged KV on SM 12.0); the patched gate reaches tml_fa4 but SplitKV shrinks tile_n to 64 against a hard 128 requirement; the hotfix’s Triton decode + SDPA prefill fallback passes numerics 10/10 and captures CUDA graphs cleanly.

The fix (hotfix-inkling-sm121-relattn.py) routes rel-bias attention on sm_121 to a numerics-verified fallback: Triton split-KV for pure decode, torch-native SDPA with the bias applied explicitly for prefill/extend/MTP. Every path validated against a hand-rolled float64 reference before the first boot - ten cases, max abs diff 1.3e-2, all pass.

The subtle part, and the reason “it worked in validation” was not the end: device-to-host reads are forbidden during CUDA graph capture. cache_seqlens.max().item() in the decode path and a .tolist() in the SDPA fallback are both invisible in eager testing and fatal under capture. The patch uses static block-table capacity while capturing and keeps dynamic reads for eager-only debugging. Result: PIECEWISE 3/3 and FULL 2/2 capture pass, and a standalone capture/replay regression shows max difference 0.0. No --enforce-eager anywhere - graphs are where the decode throughput lives.

Wall 2: the load transient, not the capacity 🔗

With the kernel fixed, the dummy-weights boot served immediately - and the real-weights boot died every time, ~15-40 seconds into shard streaming. Six boots, four configs, identical death phase.

The arithmetic looked fine at rest: ~85-90 GiB of weights per rank (159 GiB checkpoint, TP=2, dense params replicated) plus engine and host overhead lands at 102-110 GiB of the 121.7 GiB pool. Tight, servable. The killer was invisible to almost every counter:

v8 boot, 2s poller:  MemAvailable 117 -> 113 GiB  (engine init)
                     107 -> 46 GiB in ONE tick     (weight-buffer reservation)
                     46 -> 4 GiB over 12s          (shard stream fill)
                     watchdog kill at 4-5 GiB
Unevictable=0, Cached<=8, AnonPages<=8 the whole time

~110 GiB of driver allocation that only MemAvailable can see. The fill phase consumed ~42 GiB in 12 seconds while only ~16 GiB of shards had been read: the NVFP4 path on sm_121 allocates roughly 2.5x the streamed bytes as dequant/copy transient workspace. On unified memory there is no separate CPU RAM to absorb it; the pool is the pool. (This is also why Qwen3.8-Flash-Next survives on the same boxes: 67.5 GiB per rank leaves 20 GiB of margin Inkling never had.)

MemAvailable across the boot, with and without the reclaim patch. Stock loader (red): engine init, then a single reservation tick drops 107 to 46 GiB, the fill falls to 4-5 GiB and the watchdog kills the container; recovery is instant post-kill. With per-tensor madvise reclaim (green): all ten shards load in 3m34s with the pool rebounding past 30 GiB after every tensor, then the server holds steady.

The fix (hotfix-inkling-gb10-load-reclaim.py) is unglamorous: per-tensor madvise(MADV_DONTNEED) reclaim during the fill, env-gated and fail-closed. With it, the load held flat at 80.2 GiB allocated for all ten shards, 3m34s wall, MemAvailable rebounding past 30 GiB after every large tensor. The watchdog never fired.

Where it landed 🔗

vllm v0.28.0, TP=2 over RoCE (spark-4687 + spark-5bc3)
model memory     78.3 GiB per rank
KV cache         21.44 GiB -> 105,850 tokens (bf16)
graph capture    PIECEWISE 3/3, FULL 2/2
smoke            4/4 prompts, sane output

The full recipe is in the weightless repo (recipe/inkling/), and the GLP steering vector for Inkling (GLP-41, alpha=0.25 - the most sensitive dose we have measured anywhere) drops straight in, since the steering hook steers activations and does not care how the weights were loaded.

The two lessons worth keeping 🔗

  • GB10 is Blackwell-family hardware running Hopper-era assumptions. Twice now (Inkling’s FA4, GLM-5.3-Flash’s MLA routing in the community recipes) the working answer on sm_121 has been “select the SM90 kernel path, not the SM12 one.” If you are bring-up-ing anything on DGX Spark, probe the kernel dispatch table before you touch a single config knob.
  • On unified memory, watch MemAvailable, not the counters you trust. Unevictable, Cached, and AnonPages all sat near zero while the driver held 110 GiB. Capacity planning from meminfo will lie to you; watch the one number that moves, and put a watchdog on it (ours: two consecutive reads below 8 GiB kills the container, which turned every failure tonight into a recoverable event instead of a power cycle).

Steering files, patches, and the recipe are public. The model is Thinking Machines’. The two walls were NVIDIA’s and the calendar’s. Both are down.

Inkling Thinking Machines DGX Spark GB10 vLLM NVFP4 Weightless Hotfix