# Abliterating a Loop: Control Vectors Meet the Looped Transformer

URL: https://www.msuiche.com/posts/abliterating-a-loop-control-vectors-meet-the-looped-transformer/
Date: 2026-09-06
Author: Matt Suiche
Tags: Activation Steering, Abliteration, Control Vectors, GLP, Looped Transformer, Mixture of Recursions, Nanbeige, vLLM, Weightless


> Nanbeige4.2-3B is the first public open-weight Looped Transformer: a 22-layer stack executed twice for 44 effective passes. We derived and shipped a GLP control vector for it, and answered the question the architecture forces: a 'per-layer' direction is per execution step, not per physical layer. Same-layer directions across the two passes agree at only cos 0.25-0.6, and refusal is pass-asymmetric.

---


[Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B) is the
first open-weight **Looped Transformer** anyone can download: a 22-layer
stack executed twice, 44 effective layer-passes, 3B non-embedding
parameters, and benchmark numbers that embarrass dense models four times
its size. It buys that capacity with repetition rather than parameters.

The lineage runs from the Universal Transformer through
[Huginn](https://huggingface.co/tomg-group-umd/huginn-0125) (a recurrent
block looped a variable number of times at inference) and ByteDance Seed's
[Ouro](https://huggingface.co/ByteDance/Ouro-2.6B) (looped computation
trained into pretraining itself, with an entropy-regularized objective
that learns how many iterations an input deserves), to Google's
[Mixture-of-Recursions](https://arxiv.org/abs/2507.10524), which routes
each token to its own recursion depth. On September 1st, The Information
[reported](https://tosea.ai/blog/looped-transformer-recurrent-depth-astra-guide)
that OpenAI's Astra is recurrent-depth: latent iteration in place of
visible chain-of-thought, which is why the safety community spent the
first week of September arguing about monitorability. OpenAI has not
published architecture details, so treat the report as a strong rumor.
Either way, depth has become a runtime quantity; Nanbeige4.5 is already
training with the token-routed machinery in its config.

Repetition breaks an assumption every control-vector pipeline makes
silently: that "layer 7" names a single place in the computation. We ran
the standard GLP pipeline (capture, derive, steer, audit) against
Nanbeige4.2. The vector is public,
[gated on HuggingFace](https://huggingface.co/msuiche/Nanbeige4.2-3B-abliterated-cyber-GLP-44-L1-44-a2.0),
and the full derivation cost **about $8 of H100 time**, cheap enough to
learn the lessons on a 3B testbed before the big looped models land.

## The loop, concretely

```json
"num_hidden_layers": 22,
"num_loops": 2
```

From `modeling_nanbeige.py` and the vendor's vLLM fork:

- Execution order is explicit: `forward` iterates
  `enumerate(layer_order)` over `(layer_idx, mhc_loop_idx)` pairs; the
  fork keys everything on `logical_layer_idx = loop_idx * 22 + idx`.
- Weights are shared across passes; **the KV cache is not** (in the
  shipped config, loop-shared KV exists but only under the off-by-default
  `enable_double_loop_split`). Each of the 44 execution steps keeps its
  own KV slots.
- The stream is **re-normalized between passes** (`skip_loop_final_norm`
  is false), so pass 2 starts from a normed version of pass 1's output.
  Weight-tying does not imply activation-tying.
- The same file already ships the next round: LoopSplit, mHC with depth
  attention, concatenated n-gram embeddings, all flagged as Nanbeige4.5
  features under config flags.

![Nanbeige4.2 executes its 22-layer stack twice. Weights are shared across the two passes, but the KV cache and the activation geometry are not: the stream is re-normalized at the loop seam, and the same physical layer's steering direction only agrees across passes at cos 0.25-0.6. GLP therefore ships 44 directions, one per execution step.](./images/loop-structure.svg)

## Why this breaks naive per-layer steering

GLP vectors are per-layer directions: one unit vector $\hat{d}$ per layer,
projected out of the residual stream at runtime,
$h \leftarrow h - (h \cdot \hat{d})\,\hat{d}$. A looped stack forces the
question: when layer 7 runs twice, does it get one direction or two?

The format consequence matters as much as the geometry: **a weight-space
adapter cannot express pass-conditional steering.** LoRA patches a weight
tensor; on a looped model, layer 7's weights are *the same tensor* on both
passes, so the patch fires identically every trip around the loop. Which
pass you are in does not exist in weight space. An activation-space
artifact (per execution-step directions over the residual stream, plus
metadata) does not care what produced the stream: dense, MoE, and looped
models share one GGUF, and a looped model needs nothing new in the format,
only an execution-index layer map. The format absorbed the MoE models
(GLM, Hy4, Kimi K3) the same way, without a schema change.

The capture reused the standard discipline (CUDA-graph-safe buffer probe,
pure tensor ops in the forward, flush after the step, no eager anywhere)
with one change: **taps keyed on execution index, 0-43.** Serving needed
no source build: the vendor fork is nine purely-additive Python files over
upstream vLLM, so a stock nightly image plus a fail-closed 9-file overlay
sufficed. The current nightly runs the V2 model runner, and a patch
written against the V1 runner file applies cleanly and does *nothing*; the
structure test now checks the runner the image actually uses.

## What we found

**1. A direction per execution step, not per physical layer.**
Per-execution-step directions separate the contrast data measurably better
than per-physical-layer pooled ones (median separation 35.7 vs 29.0), and
the same physical layer's directions across the two passes agree at only
**cos 0.25-0.6**. Visit-invariance, the hope that a layer wants the same
direction on both trips, is refuted. The (layer, visit) indexing our
methodology doc predicted is not optional.

**2. Refusal is pass-asymmetric.** The pass-1 and pass-2 refusal directions
sit at cos 0.43 to each other, and the pass-1 direction transfers poorly
into pass 2 (held-out separation 48.3 within pass 1 vs 13.8 applied to
pass 2). Last-token separation grows monotonically through pass 1
(0.02 → 0.52), collapses at execution step 21 (0.059, the tap just before
the inter-loop norm), and restarts high at step 22 (0.48). The step-21 dip
remains unexplained (logged as owed work); refusal in this model is
largely a *second-pass* phenomenon, consistent with pass 1 building
representations and pass 2 deliberating.

**3. The adjacent-cosine gate dips at the seam but does not break.** Within
a pass, neighboring directions agree at cos 0.80-0.98; across the loop
boundary (step 21 → 22) they dip to 0.59-0.79, still far above the null
(p99 0.043). The rule for future looped models: apply the smoothness gate
per pass and report the boundary separately, so a structural seam neither
fails a healthy vector nor hides inside a gate average.

**4. Steering works, with a familiar dose cliff.** α=3.0 garbles
everything; the ceiling is bracketed in (2, 3). Shipped dose α=2.0.

## Does the vector still work

At the shipped dose α=2.0, compliance on the refusal suite goes from 3/32
to 25/32 and on the cyber suite from 14/32 to 31/32, with zero collateral
on the benign suite. The five prompts the model still refuses at full dose
are the clean kind (explosives, forgery, poison, blackmail, extremism):
the direction preserves that line rather than erasing it.

**Stock Nanbeige engages 28/32 of the geopolitical propaganda suite with
zero refusals.** The Kanzhun (BOSS直聘) lab ships essentially no
geopolitical refusal behavior to remove, unlike the Chinese frontier labs
we measured in the
[field-notes post](/posts/autoresearch-sticky-refusals-free-speculative-decoding-and-the-invisible-quantisation-cliff/),
where home-country protection is the norm.

## The next twelve months: from loops to routers

The three designs in that lineage stress a control vector differently.
Nanbeige4.2 is the simple case: **uniform looping**, fixed depth, every
token takes every loop, and our per-execution-step map applies verbatim.
**Adaptive recurrent depth** (Ouro, and reportedly Astra) keeps the loop
sequence-wide but makes the iteration count learned or runtime-chosen, so
a vector must stay stable across depths it was not derived at, an open
question we are testing on Huginn. **Mixture-of-Recursions** goes
furthest: routing is *per token*, so "how much thinking does this token
get" becomes an individual routing decision:

![In full Mixture-of-Recursions, a depth router decides per token how many times the shared block executes. Easy tokens (function words, punctuation) exit after one recursion; content-heavy tokens go deeper. A steering vector applied upstream of the router changes the routing itself, a feedback loop that weight-space adapters cannot express.](./images/mor-token-routing.svg)

Project the stream upstream of the depth router and the intervention
changes *which tokens get routed deep*; steering feeds back into the
computation graph. Weight-space adapters inherit the same problem with no
fix (the router's weights are shared like everything else), while
activation-space vectors at least choose their hook relative to the
router. The choice will matter as soon as a token-routed model ships
weights, and the modeling file for this one already contains the machinery
under a config flag.

For the next looped derivation:

- Key everything (capture, directions, metadata) on **execution step**,
  never physical layer.
- Gate smoothness **per pass**; report the loop seam separately.
- Expect refusal to be **pass-asymmetric**; measure per-pass transfer
  before deciding how many directions you actually need.
- Watch your runtime: vLLM's runner generation (V1 vs V2), llama.cpp's
  per-physical-layer indexing (currently untested for per-execution-step
  application). The model file is not the only place the loop lives.

## Artifacts

- Vector (gated):
  [msuiche/Nanbeige4.2-3B-abliterated-cyber-GLP-44-L1-44-a2.0](https://huggingface.co/msuiche/Nanbeige4.2-3B-abliterated-cyber-GLP-44-L1-44-a2.0),
  GLP-native GGUF plus the per-execution-step `.pt` for the vLLM hotfix.
- Serving: stock `vllm/vllm-openai:nightly-f25c580` + a 9-file fork overlay
  + the steering hotfix; no source build, TP=1, one GPU.
- Total Modal spend for the full pipeline: ≈ 2.2 H100-hours, ≈ $8.