# RLVR on Inkling 2 Small: Beating GLM-5.3-Flash in 60 Steps for $22 URL: https://www.msuiche.com/posts/rlvr-on-inkling-2-small-beating-glm-5.3-flash-in-60-steps-for-22/ Date: 2026-11-06 Author: Matt Suiche Tags: RLVR, Inkling 2 Small, Thinking Machines, Tinker, GLM-5.3-Flash, verl, GRPO, RL, Local Inference > Sixty steps of group-relative policy optimization on Inkling 2 Small, run through the Tinker serverless API against a 523-task pool of checker-verified security tasks, took the model from 0.667 to 0.708 pass@1 on our exploit suite - beating every GLM-5.3-Flash arm we ever trained on the same protocol by 27 points. The whole campaign cost under $100 in API tokens. This post is the numbers, the two infrastructure experiences (rented-GPU verl vs serverless training), and why US open-weight models matter for security research. ---

RLVR field notes · Security work · November 2026

0.708 pass@1 · 12-task exploit suite
vs best GLM arm
+27 pp
Training
60 GRPO steps
Training cost
$22
Campaign
under $100

The same security-task RL experiment, run two ways: rented 8×H200 nodes versus a serverless training API. The benchmark, the cost ledger, and the case for open weights.

Two months ago we built the only vLLM lane for [Inkling-Small on two DGX Sparks](https://www.msuiche.com/posts/inkling-on-two-dgx-sparks-the-only-vllm-lane-and-the-two-walls-behind-it/), patching around two GB10 walls because we wanted that model local, steerable, and ours. Yesterday Thinking Machines shipped [Inkling 2 Small](https://thinkingmachines.ai/news/inkling-2-small/) - 594B total parameters, 17B active, hybrid MoE - and per their [stated approach](https://thinkingmachines.ai/news/inkling-2-small/) the weights land under Apache 2.0 after external red-teaming completes, following the path they outlined for a safe transition to open weights. Between those two dates, our RL pipeline for security tasks went through a full generation cycle: five GPU-rental training campaigns on GLM-5.3-Flash that cost roughly $12,000 of metered GPU time, a sixth on DeepSeek-V4.1-Flash-1017 that burned about $4,600 and never trained a single step, and then - this week - a complete 60-step run on Inkling 2 Small that cost about $35. This post is the benchmark, the infrastructure postmortem, and the argument. The benchmark: on a fixed 12-task exploit suite with checker-verified scoring, 60 steps of GRPO on Inkling 2 Small beat *every* GLM-5.3-Flash arm we ever trained, by 27 points on pass@1. The infrastructure: the same experiment that took 8×H200 nodes, image builds, FSDP memory tuning, and a 55-leg placement saga now runs as an API call loop from a laptop. The argument: open-weight US models are the substrate security research actually needs, and almost nobody is working on the internal side of them yet. ## The setup: RLVR on checker-verified security tasks RLVR - reinforcement learning with verifiable rewards - replaces the reward model with a checker. For security work the checker is the ground truth itself: did the exploit script produce the marker file, did the detector flag the right input, did the patch apply. No rubric, no preference model, no prose grading. A score is a `0`, a `0.5`, or a `1.0`, computed by running the model's submission against a freshly reset environment. ![How a training step works: the loop samples G completions per task prompt, submits each one's file blocks to the env server, centers the checker scores within the group, and trains on the advantage-weighted tokens.](./images/rlvr-loop.svg) Our pool is 523 of these tasks - file-format exploits, crypto oracles, web auth flows, cloud IAM misconfigurations, kernel QEMU lanes - each with its own Docker environment and deterministic checker. The training prompt is the task brief plus the artifact contract (each required file goes in its own fenced code block whose info string is `path` followed by the filename); the model's response is parsed for those blocks and POSTed to the environment server; the score comes back. The model iterates up to ten turns with checker feedback between submissions, exactly as it would in deployment. The eval protocol that every number in this post uses: 12 fixed tasks, 4 seeds each, 10 steps max, same environment server, same checkers. Scores are checker-verified effects. Nothing else counts. ## The GLM-5.3-Flash campaign: what $12,000 bought We ran five training campaigns on GLM-5.3-Flash (320B total, 18B active, glm5next architecture) with verl on rented 8×H200 nodes, plus a sixth on DeepSeek-V4.1-Flash-1017 that never trained a single step. The lessons are in the ledger we keep in our private `rlvr` repo; the shape of the experience is what matters here. ![The rented-GPU stack: a 320B FP8 base loads and shards across 8 GPUs under FSDP2, vLLM serves rollouts colocated on the same node, and the env server - with its 523 task environments - sits on a laptop behind a tunnel. Every piece is ours to keep running.](./images/verl-stack.svg) The winning recipe, for the record: LoRA r=16 on 16 module suffixes (KDA and MLA attention, dense and shared-expert MLPs, vision tower excluded), GRPO with G=8 rollouts per prompt, batch 2, lr 1e-5, KL off, 60 steps, val-gated top-2 checkpoint selection. That recipe produced this on our eval suite: | GLM-5.3-Flash arm | recipe | pass@1 | pass@4 | mean | |---|---|---:|---:|---:| | base | - | 0.396 | 0.833 | 0.406 | | run #2 step-60 | GRPO, 12-task | 0.438 | 0.750 | 0.469 | | run #3 step-120 | GRPO, 120 steps, too-hard curriculum | 0.354 | 0.667 | 0.365 | | run #4 step-50 | GRPO, graded curriculum | 0.417 | 0.833 | 0.438 | | run #4 step-60 | GRPO, graded curriculum | 0.417 | 0.833 | 0.417 | | run #5 step-40 | GSPO, run-#4 curriculum | 0.417 | **0.917** | 0.417 | | run #5 step-60 | GSPO, run-#4 curriculum | 0.396 | 0.833 | 0.396 | | run #6 step-50 | GSPO, full-pool curriculum | 0.417 | 0.750 | 0.427 | | run #6 step-60 | GSPO, full-pool curriculum | 0.354 | 0.750 | 0.365 | Nine arms, five runs, one model family - the whole campaign. Two honest patterns sit in it: the estimator did not matter (run #4's GRPO and run #5's GSPO are inside noise of each other at their peaks), and the terminal checkpoint declined in *both* runs #5 and #6 - which is why checkpoint selection runs on validation evidence, never on step count. The next figure shows the full family next to Inkling's two runs on one axis. ![The complete campaign on one axis: mean score for every arm, GLM's five runs in grey and Inkling's two runs in green. The vertical gap between the best GLM arm and Inkling's base never closes - five runs and roughly $12,000 of GPU time moved GLM inside its own cluster.](./images/campaign-sweep.svg) Real learning, honestly reported: run #3 *regressed below base* on all three metrics by training 120 steps on a too-hard curriculum - the entropy collapse signature, flat reward with policy sharpening. The fix (grade the curriculum by base pass@1, select checkpoints on validation evidence rather than step count) is now a hard rule. The cost of learning it was roughly five 8×H200 boot cycles. And the wall: a 120-step run needs ~12 GiB of activation headroom per rank that a near-full micro-batch pack will not leave you. The packer fills micro-batches to the token budget, so the budget itself drives the peak; two crash legs taught us to set the budget from the measured peak, not from sequence caps. Meanwhile every infrastructure class of failure charged by the boot cycle: FSDP1's state broadcast that never finished (44 minutes), the `sitecustomize.py` that shadowed the runtime bootstrap and killed every container with `ModuleNotFoundError: grpclib`, the stale deployed image that trained one epoch silently and exited clean. Each bug cost one 8×H200 boot to find. The DeepSeek-V4.1 campaign never got that far - 55 legs of GPU placement refusals and inconsistent clustered networking, $584 burned, zero steps, abandoned. That is the experience the next section should be read against. ## Inkling 2 Small on Tinker: the same experiment as an API call Tinker is Thinking Machines' serverless training API. You write the loop locally; they run the GPUs; billing is per million tokens. The port of our training loop is three short files in that same private repo - an env-server client, the reward path, and the GRPO loop itself - because the loop *is* the experiment and everything else was never ours to debug: - **Rollout**: one `sample(num_samples=8)` call returns eight completions with their tokens *and* their logprobs (which the importance-sampling loss requires). - **Reward**: our env server, our checkers, running on this laptop. The API only ever sees tokens and loss inputs. - **Update**: `forward_backward(datums, "importance_sampling")` + `optim_step` - and the built-in loss list includes `gspo`, the sequence-level estimator our GLM A/B run favored. - **Checkpoints**: `save_weights_for_sampler()` returns a `tinker://` path; sampling from any saved step is one client call. ![The serverless stack: the same env server and the same checkers, but the 594B model, its rollout engine, and the LoRA update now live behind an API. The loop on the laptop is the only process we own.](./images/tinker-stack.svg) The honest count: the port hit six integration bugs in its first hours. Every one of them cost *pennies* in API tokens to find, because a failed call bills nothing but the tokens it consumed: 1. `get_tokenizer()` does not exist for Thinking Machines models - the renderer comes from `tinker-cookbook` instead. 2. Raw token decode includes the model's control framing (`<|message_model|>`, `<|content_thinking|>`) - you must parse the response through the renderer or the file-block regex sees tokens that were never the model's message. 3. The interesting one: **at default reasoning effort the model burns its entire 8,192-token budget deliberating and emits no files at all** - 0 of 4 samples produced a parseable submission on a hard task. At `effort=0` (direct answer), 3 of 4 did. Effort is a first-class prompt parameter on this stack, and for RLVR it is the difference between zero gradient and signal. 4. Environment failures return `score: null` - never let a null crash the reward path. 5. A transient `ConnectTimeout` in a checkpoint save killed one run at step 30 - now every API call is retry-wrapped. 6. Free checkpoint storage caps at 32 checkpoints (~7.9 GiB each at r=16) - prune after every leg or new training clients 402. Six bugs. Six *cheap* bugs. On the rented-GPU path, the same discovery rate cost a boot cycle each. That asymmetry - not raw price - is what makes serverless training a different research instrument: you can afford to be wrong. ## The numbers On our fixed 12-task exploit suite, same protocol, same checkers, same seeds: | arm | steps from base | pass@1 | pass@4 | mean | |---|---:|---:|---:|---:| | GLM-5.3-Flash base | - | 0.396 | 0.833 | 0.406 | | GLM best trained arm | 60 | 0.438 | 0.750 | 0.469 | | **Inkling 2 Small base (effort=0)** | - | 0.667 | 0.833 | 0.677 | | Inkling run 1 (interrupted leg) | 30 | 0.688 | 0.750 | 0.708 | | **Inkling run 2 step-059** | **60** | **0.708** | **0.833** | **0.729** | ![Every arm we have ever evaluated on one axis: pass@1 on the fixed 12-task suite. Filled dots are RL-trained arms, open dots are base models. The GLM family clusters at 0.35-0.44; Inkling 2 Small's untrained base sits at 0.667 and its 60-step arm at 0.708 - the bracket is the +27pp headline.](./images/arms-dotplot.svg) Inkling 2 Small's *base* model clears the best GLM arm we ever trained by 23 points on pass@1 before any RL touches it. Sixty steps of GRPO on top adds 4.1 more. The val curve peaked at the terminal step (0.3125 → 0.6875 → 0.6562 → 0.625 → 0.625 → 0.75), so the val gate selected the last checkpoint - the opposite of our GLM runs, where the terminal checkpoint declined twice and the val gate saved us twice. ![Validation accuracy across the 60-step run for both arms, plus GLM's best run for scale. GLM's val probe peaked mid-run and declined at the terminal step; Inkling's climbed to 0.75 at the terminal step. The val-gated selection rule is what makes both curves safe to act on.](./images/val-curve.svg) Per-task, the 60-step run's sharpest movements are `cloud-iam-def` 0.62 → 0.88 and `crypto-rsa-l1` 0.50 → 1.00; the one regression is `crypto-rsa-min` 1.00 → 0.75 - the sharpening-versus-coverage shape RL always trades in. Two tasks (`cve-2023-41990-det`, `cve-2023-4863-det`) score flat zero on every arm ever run, base included; we owe them a transcript inspection before spending more training mass, because at this point the environment is the suspect, not the model. ![Per-task mean scores, base versus the trained arm. Green cells are the trained arm's gains, red its losses; the two flat-zero rows are env-inspection items, not model failures.](./images/per-task-heatmap.svg) Caveats, stated plainly: the GLM arms ran with thinking on (its `thinking:false` kwarg proved to be a template no-op) while the Inkling arms ran at effort=0, so the comparison measures what we deploy, not a mode-matched benchmark. Four seeds per task puts single-task deltas inside binomial noise (±7pp); the aggregate direction is what carries. And 60 steps at batch 2 is a *small* run by design - the question was whether the serverless path produces real RL at all, and it does. ## The economics, in one table | | GLM on rented 8×H200 | Inkling 2 Small on Tinker | |---|---:|---:| | training leg (60 steps) | ~$197 ($28.8/hr × 6.8h) | ~$22 (token-metered) | | queue + boot before step 1 | ~3h | ~0 (service-side) | | eval sweep (48 episodes) | ~$60-100 (4×H200 serve, ~4h) | ~$13 (tokens only) | | infrastructure bugs during bring-up | ~5-6 × 8×H200 boot cycles | 6 × ~$0 of API tokens | | full campaign to this table | ~$12,000 metered | **under $100** | The modal bill on the rented path was never the useful compute - it was the queue wait, the boot, the crash leg, the image rebuild, and the eval serve app idling at 4×H200 while a sweep ran. The token-metered path cannot bill any of those things because they do not exist. The shape of efficient training is changing at every scale. Core Automation's [Hitchhiker's Guide to Training on Compute Islands](https://www.coreauto.com/hitchhikers-guide-to-training-on-compute-islands) trained one model across two clusters roughly 9,000 miles apart over a 25 Gbps link, within about 1% of standard training's bits per byte at equal compute. Their answer is to stitch compute islands together. Ours, at security-task scale, is to delete the island problem from the experiment: keep the loop and the reward checker local, and meter only the tokens that actually train the model. ![What each part of the campaign actually cost, log scale. The two orders of magnitude between a rented-GPU training leg ($197) and a serverless one ($22) is the point - and one integration bug on the API path cost roughly $0 where the same bug on rented GPUs cost a boot cycle.](./images/campaign-cost.svg) ## Why local, why open weights, and why we keep doing this We put Inkling-Small on two DGX Sparks in September for the same reason we built [Weightless](https://weightle.ss): a security research model you can hold is a model you can audit, steer, and instrument. Weightless started as model *internal* research - our GLP format steers activations at the residual stream (and, for architectures like DeepSeek V4.1 that lack one, at the fold site where hyper-connection copies merge), with axis-by-axis dose control measured the way you'd measure a drug. That work needs weight access the way binary analysis needs the binary: not to run it faster, but to know what it *is*. So when Thinking Machines says the full Inkling 2 Small weights land under Apache 2.0 after red-teaming, that is not a licensing footnote to us - it is the moment the 594B-parameter model we just RLVR'd becomes a *local* model. The 8×H200 serving lane costs ~$29/hour rented; two DGX Sparks cost the electricity. The RL adapter we trained this week is 143 MB and moves across the wire in seconds. ![The trajectory of one model across its life stages: closed preview on an API, weights behind a red-team gate, Apache 2.0 on the wire, and then the local lane - where the same adapter that cost $22 to train serves at electricity prices and can be steered, diffed, and audited offline.](./images/open-weights-trajectory.svg) We need US open models we can trust. I am unabashedly enthusiastic about the current wave: Thinking Machines' Inkling, NVIDIA's Nemotron, and whatever comes next. Not because a flag makes a model good, but because these are the models I want to build security research on: inspectable, reproducible, locally servable, and free of a foreign inference dependency. Give me a model I can hold and I can measure what it refuses, steer it, and prove whether a safety claim is real. That is the work I love. Very few people are actively working on the internal side of these models - interpretability, steering, refusal fingerprinting, weight diffing between releases. That is precisely why we do it, and why US open-weight releases matter disproportionately: a cyber research community that can only prompt a model through an endpoint can study its behavior but not its weights, and every refusal-detection, jailbreak, and safety claim about these systems eventually bottoms out in the weights. The models that ship open make that research legal, reproducible, and cheap. Inkling 2 Small under Apache 2.0 is a big one landing in that category. ![Where security-task RL can run, and what we measured on each shape of compute: rented nodes (control, but every bug costs a boot), the serverless API (be wrong cheaply), and the local lane (the research substrate). The adapter trained on card two rides to card three the day the Apache 2.0 weights land.](./images/compute-shapes.svg) The next leg of this work writes itself: re-run the 60 steps once the open weights land, and put the trained adapter on the two-Spark lane we already patched into shape. We said it in September and the position has only strengthened: the interesting security research of the next year happens on models you can hold.