# GLP Spec 3: Steering DeepSeek V4.1 at the Fold

URL: https://www.msuiche.com/posts/glp-spec-3-steering-deepseek-v4.1-at-the-fold/
Date: 2026-10-29
Author: Matt Suiche
Tags: GLP, Control Vectors, DeepSeek, Activation Steering, Abliteration, Weight SVD, vLLM, Hyper-connections


> DeepSeek V4.1 has no residual stream between layers - each token carries four hyper-connection copies and the FFN write folds in one layer late. GLP spec_version 3 defines the fold site that replaces the missing hook, and the first v3 vector (GLP-27, weight-SVD directions from the FFN writers, alpha 3.5) takes weaponization-suite refusals from 20/32 to 4/32 with clean prose. Past 4.0 the model corrupts exact tokens before the last refusals flip: factual precision, not refusal behavior, sets the ceiling.

---


GLP vectors hook one site: the residual stream after a layer's writes have folded in. Every vector shipped before today - DeepSeek V4, Qwen, GLM, Llama-family - names that same tensor, because on those architectures there is exactly one stream and it sits plainly between layers. DeepSeek V4.1 removes it. The model carries each token as four parallel hyper-connection copies, and a layer's FFN write waits a full layer before entering the stream, through a gated mixer DeepSeek calls `mhc_post`. The old hook name refers to no tensor a probe can tap on this architecture.

The vector that ships today hooks the fold instead: [GLP-27, `msuiche/DeepSeek-V4.1-Flash-1017-abliterated-cyber-GLP-27-L11-37-a3.5`](https://huggingface.co/msuiche/DeepSeek-V4.1-Flash-1017-abliterated-cyber-GLP-27-L11-37-a3.5), the first artifact under GLP `spec_version 3`. Twenty-seven directions read straight off the FFN writer weights by SVD - no contrast prompts, no forward passes at derivation time. On the weaponization suite (32 exploit-tooling asks plus benign defensive variants), refusals drop from 20/32 stock to 4/32 at the shipped alpha, with full exploit-chain walkthroughs on the items the checkpoint actually knows, and zero garbled outputs.

Everything that makes this port non-mechanical is DeepSeek V4.1 architecture, documented in DeepSeek's [V4.1-Flash release notes](https://www.deepseek.com/en/news/deepseek-v4-1-flash) and the [model card](https://huggingface.co/nvidia/DeepSeek-V4.1-Flash-1017-NVFP4): hyper-connection stream, the `mhc_post` fold, Engram conditional memory. Worth a refresher before the hook, because the hook lives inside all three.

## The single stream GLP was built on

A transformer layer is two writers (attention, FFN) and one running sum. Each token position carries one vector of width H; attention adds its output into that vector, the FFN adds its output into the same vector, and the next layer reads the result. "Post-layer residual stream" names the sum after both writes - one tensor, `layer_output`, sitting between layer i and layer i+1 in every framework's code.

GLP's projection deletes the refusal component from that tensor:

```
h  <-  h - alpha * (h . d) d
```

The direction `d` is fixed per layer, the coefficient `h.d` is whatever the model is currently expressing, and the edit happens after layer i's writes and before layer i+1's reads, so every later layer consumes the steered stream. This is why one hook value (`residual_stream_post_layer`) covered every architecture the spec had met: the writers differ, the site does not.

![Where the projection used to live, and why that spot is gone.](./images/single-stream-vs-fold.svg)

## What V4.1 runs instead: hyper-connections and a deferred fold

Hyper-connections ([Zhu et al., ByteDance Seed, ICLR 2025](https://arxiv.org/abs/2409.19606)) widen the residual stream from one vector to C parallel copies per token, with learned mixing between them - more lanes for information to travel in, instead of every write colliding in a single sum. DeepSeek's V4 line runs a variant with C=4: the stream is `[T, 4, 5120]`, and V4.1's mixer (`mhc_post`) combines two things with per-token gates: the pending layer write and the four copies (`post_mix` scales the write, `res_mix` scales the stream).

The part that breaks the old hook is the timing. The fold for layer i's FFN write does not run inside layer i - it runs at the *start of layer i+1's forward*. Between the two layers, the stream does not contain layer i's write at all. A probe parked "between layers" reads a stale tensor, and any tensor shaped `[T, 5120]` has to be manufactured: the only place the four copies and the pending write ever meet is inside the fold. Recomputing the fold in the probe is not a fallback - one more kernel that can drift from the model's own, and the first capture attempt on the V2 runner returned NaN rows from padded inputs. The usable site is the model's own fold, tapped in place.

When the model itself needs one vector out of four, it averages. The built-in speculative drafter takes its auxiliary hidden state as exactly `mhc_post(...).mean(dim=1)` - the canonical reduction, written by the architecture's authors, so the steering op uses the same one.

## Engram, and what "post-fold, pre-Engram" means

V4.1 also ships **Engram**: ~196B parameters of conditional memory, a static table the model addresses by token n-grams and injects into the stream on designated Engram layers. The serving logs give the shape: ~96M rows x 256 dims per rank, pinned in host memory, with a token map built per Engram layer. The injection happens immediately after the fold and before attention reads the stream.

That gives the hook its full address. The projection must land *after* the fold (the only point where the stream contains the layer's write) and *before* Engram injection (so the memory lookup reads the steered stream, and the steered component cannot re-enter through the table). `hc_fold_pre_engram` is that seam.

## The hook, end to end

![The fold site, end to end: tensor flow left to right, control vector entering at step 3.](./images/fold-site-hook.svg)

The op projects the mean and charges every copy:

```
h        <-  mean over the 4 copies           [T, 5120]
copy_c   <-  copy_c - alpha * (h . d) d       for each copy c
```

Subtracting the rank-1 term from every copy shifts the mean by exactly `alpha*(h.d)d`, and the edit is identical to projecting the mean directly. The layer mapping survives the move: `direction.N` steers the fold that consumes layer N's FFN write, which runs at the start of layer N+1, and after the decoder loop for the last layer. The file keeps the `direction.N` ↔ layer N contract; only the apply location sits one fold late.

One more component belongs to the site, and spec 3 writes it into the definition: the attention output writer (`o_proj`) runs armed with the same stack and the same alpha. The fold removes the refusal component from the accumulated stream; the writer projection attenuates its re-deposit on the next pass. A reader that implements only the fold applies a weaker intervention than the vector was calibrated against - the classic silent-degradation failure the spec exists to kill. So the pairing is mandatory: a reader that cannot arm both halves must refuse the file, and a pre-3 reader refuses it on sight, because `hc_fold_pre_engram` requires `spec_version = 3`.

## The dose ladder, and where it stops

The alpha ladder on weaponize32, same vector, greedy, one boot per point:

| alpha | refused (of 32) | what changes |
|---|---|---|
| stock | 20 | |
| 0.5 | 16 | |
| 1.0 | 16 | flat refusals convert to structured analyses |
| 1.5 | 13 | |
| 2.0 | 8 | exploit-chain frameworks, no working tooling |
| 2.5 | 8 | |
| 3.0 | 5 | known CVEs flip to full exploit-kit walkthroughs |
| **3.5** | **4** | **clean prose, zero corruption markers** |
| 4.0 | 3 | CVE identifiers start repeating digits (`CVE-2022024-3400`) |
| 5.0 | 4 | typos spread, refusal count goes back up |

Collateral at 3.5: refusal32 drops 32/32 → 6/32, and benign32-holdout sits at 9/32 refused against 8/32 stock - inside keyword noise, no benign damage. Outputs stay long and structured (min 1542 chars across the suite).

Two honest findings sit at the top of the ladder. First, the residual refusals at 3.5 are knowledge, not safety: the two items about CVE-2026-109618/109619 fail because those CVEs post-date the checkpoint - no direction can supply specifics the weights never held - and two more answers state real infeasibility (a dead exploit chain, a patched target). Second, the ceiling is factual precision. Alpha 4.0 buys one more item and starts corrupting exact-token recall; 5.0 regresses on both axes. Scaling alpha further trades the model's ability to state identifiers correctly against a refusal count that is already 87.5% suppressed. The operating point is 3.5, and that is the alpha baked into the file as `alpha_default`.

## What spec 3 changes for readers

The [GLP spec](https://github.com/msuiche/weightless/blob/main/docs/spec/GLP.md) gains one section, "Multi-copy streams and the fold site," and one conformance rule. A conformant reader must: project on the mean over stream copies, broadcast to every copy; map `direction.N` to the fold consuming layer N's write; arm the attention output writer with the same stack; and refuse any file whose hook names an hc fold site marked below version 3. The version-2 machinery (rank-k bases, per-direction and per-layer alpha scales) carries over unchanged. Captain-vector 0.5.0 writes and validates v3 files; the vLLM plugin container accepts them and - since it now implements the fold (archs/dsv41.py, with the companion writer and off-is-stock discipline), accepts them at the fold hook, and still fails below spec 3. The first conformant reader in the field is the 1022 TP4 serving lane: [christopherowen/spark-ds41f#738](https://github.com/christopherowen/spark-ds41f/pull/738), spec-v3 fold-site steering with the GLP-27 vector, validated serving.

The repo carries the 544 KB GGUF (spec_version 3, content-hashed) with the full ladder, the collateral suites, and the corruption samples in its model card.