Abliterating a Loop: Control Vectors Meet the Looped Transformer

Sun, 06 Sep 2026 00:00:00 +0200

Nanbeige4.2-3B is the first open-weight Looped Transformer anyone can download: a 22-layer stack executed twice, 44 effective layer-passes, 3B non-embedding parameters, and benchmark numbers that embarrass dense models four times its size. It buys that capacity with repetition rather than parameters.

The lineage runs from the Universal Transformer through Huginn (a recurrent block looped a variable number of times at inference) and ByteDance Seed’s Ouro (looped computation trained into pretraining itself, with an entropy-regularized objective that learns how many iterations an input deserves), to Google’s Mixture-of-Recursions, which routes each token to its own recursion depth. On September 1st, The Information reported that OpenAI’s Astra is recurrent-depth: latent iteration in place of visible chain-of-thought, which is why the safety community spent the first week of September arguing about monitorability. OpenAI has not published architecture details, so treat the report as a strong rumor. Either way, depth has become a runtime quantity; Nanbeige4.5 is already training with the token-routed machinery in its config.

Nanbeige on Matt Suiche

Abliterating a Loop: Control Vectors Meet the Looped Transformer