Intelligence Ladders: One Set of Weights, Every Depth a Model

Needle 3 runs on watches, Raspberry Pis and phones. Each device has a different memory budget, so each wants a different model size. Training one model per size means several training runs, several files and a separate fine-tune for each. Needle 3 is one set of weights. Every depth from 2 to 20 layers is a trained model, and all of them run on the same engine. We call this an intelligence ladder.

Blocks and the residual stream

A transformer is a stack of blocks. Needle 3 has twenty, $f_0, \dots, f_{19}$. They share one running vector, the residual stream. Block $\ell$ reads the stream, computes an update, and adds it:

$x_{\ell+1} = x_\ell + f_\ell(x_\ell).$

Because each block only adds to the stream, skipping a block means adding nothing, the same fact that stochastic depth and LayerDrop use during training. The stream passes through unchanged and the model is simply shallower. Two questions remain: which blocks to skip, and how to make the shallow models good.

Which blocks make a subnetwork

A subnetwork of depth $d$ is a set $S_d \subseteq \{0, \dots, 19\}$ of blocks, run in their original order. The sets must nest, $S_2 \subset S_3 \subset \cdots \subset S_{20}$, so that one set of weights serves all of them. The chosen blocks must also stay spread through the stack, so that a shallow model sees early and late computation rather than only the first few blocks.

One rule gives both. Start from the two ends, then repeatedly split the widest gap between chosen blocks at its midpoint:

$S_2 = \{0,\ L-1\}, \qquad S_{d+1} = S_d \cup \Big\{\Big\lfloor \tfrac{a+b}{2} \Big\rfloor\Big\}, \quad (a, b) = \arg\max_{\substack{a<b \text{ adjacent in } S_d}} (b - a),$

with ties going to the leftmost gap. Each step only adds a block, so the sets nest. Each step halves the largest hole, so the coverage stays balanced.

Twenty blocks give one fixed sequence, $0, 19, 9, 14, 4, 6, 11, 16, 2, 7, 12, 17, 1, 3, 5, 8, 10, 13, 15, 18$. The subnetwork of depth $d$ is its first $d$ entries.

Running a subnetwork

With $\mathbb{1}[\ell \in S_d]$ equal to 1 when block $\ell$ is in the set and 0 otherwise, the forward pass at depth $d$ is

$x_{\ell+1} = x_\ell + \mathbb{1}[\ell \in S_d]\, f_\ell(x_\ell), \qquad \ell = 0, \dots, L-1,$

followed by the same final normalisation and the same output layer as the full model.

Three details make this hold in the real architecture:

• Needle's residual stream has four parallel lanes, and blocks read and write them in round-robin order. Inside a subnetwork the order is renumbered over the blocks that remain, so consecutive surviving blocks still alternate lanes.
• The engram memory sites and the global-attention layers belong to specific blocks. They are kept when their block is kept and dropped when it is dropped.
• The KV cache is per block, so a shallower model also has a proportionally smaller cache.

Training every depth at once

Adding one loss term per depth to every training step would cost $L - 1$ forward passes per step. Instead, each step samples one path. With probability $0.8$ it is the full model and the step is an ordinary language-modelling step. Otherwise a depth $d$ is drawn uniformly from $\{2, \dots, L-1\}$, only the blocks in $S_d$ run, and the loss is scaled by the fraction of the model that took part:

$\mathcal{L}_{\text{step}} = \begin{cases} \mathrm{CE}(p_L) & \text{with probability } 0.8,\\[4pt] \dfrac{d}{2L}\Big[\mathrm{CE}(p_d) + \lambda\, \mathrm{KL}\big(\operatorname{sg}[p_L]\,\|\,p_d\big)\Big], \quad d \sim \mathcal{U}\{2, \dots, L-1\} & \text{with probability } 0.2. \end{cases}$

$p_d$ is the next-token distribution at depth $d$, $\mathrm{CE}$ is cross-entropy against the true next token, and $\operatorname{sg}$ is a stop-gradient: the full model's prediction is a fixed target on that step, and no gradient flows into it.

The KL term does most of the work for the shallow depths. Cross-entropy asks a 3-layer subnetwork to predict the next token, which it can barely do, and its gradient is small. The KL term asks it to predict what the whole network predicts. The full model is the teacher, each subnetwork is a student, and because they share every weight, the teacher costs no extra forward pass. This is self-distillation, with $\lambda = 1$.

Each step trains one depth. Averaged over many steps, this gives the same expected loss as training every depth on every step, and it costs one forward pass instead of eighteen.

The ladder also improves the full model. We trained the same 18-layer architecture twice for 3,000 steps on the same data, once with the ladder and once without. With the ladder the full-depth validation loss was 1.958; without it, 2.037. The ladder model was ahead at every checkpoint, and its 16-layer exit alone matched the control's full 18 layers. Every prefix of the stack has to predict well on its own, so the early blocks cannot leave the work to the later ones, and no single block can become one the model depends on.

The numbers behind the figure. Bits per byte is the validation text loss in a unit that does not depend on the tokenizer; lower is better.

Two things follow. The ladder costs the full model nothing and gains it 0.079 nats, 3.9% of the loss. And the ladder has to be there from the start: switching it on late gives worse subnetworks at every depth and a slightly worse full model, 1.1% above the control, because the blocks have already specialised around a fixed depth.

What each depth of the ladder-trained model keeps and what it costs, against its own full depth. The parameters kept are the surviving blocks with the engram sites and the confidence rows that belong to them; the embeddings and the final layers are shared by every depth.

Down to 4 layers the loss grows more slowly than the parameters shrink. At 2 layers it more than doubles.

What each depth scores

Every depth from 2 to 20 layers ships in the one Needle 3 file, and needle build --layers picks any of them. We benchmarked four of them on two tool-calling suites, sliced from the same checkpoint and run through the same engine with the confidence gate on.

The 2-layer subnetwork is absent because at 2 layers the base model's confidence head withholds almost every call, so its gated score is zero, and with calls forced it scores 20.5 on DroidCall. Two layers of a generalist is not a product. Two layers fine-tuned on one product's tools is, which is the next figure.

Fine-tuning lifts every depth by 18 to 36 points on DroidCall, and from 4 layers up the tuned subnetwork passes DeepSeek V4 Flash. The 2-layer model goes from unusable to 56.5. A small network trained on one narrow task reaches frontier accuracy on that task, and the ladder is what lets you choose how small.

Fine-tune once, deploy any depth

Because the subnetworks share weights, one fine-tune is a fine-tune of all of them. The LoRA adapter is trained on the frozen base at the full 20 layers, merged into the weights, then sliced:

$\theta_d = \mathrm{slice}\big(\theta + BA,\ S_d\big),$

where $BA$ is the low-rank update and $\mathrm{slice}$ keeps the blocks in $S_d$, the engram tables of their sites, and the rows of the confidence head that belong to them.

A sliced model has its own block numbering, 0 to 15 for a 16-layer slice. Running the bisection rule on that numbering picks a different set of blocks from the parent's, blocks that were never trained together as a subnetwork. Of the 14 depths of a 16-layer slice, 13 differ between the two rules.

So every slice carries its parent's selection order, and slicing a slice equals slicing the original. In the Python package:

needle finetune data.jsonl --epochs 10 --out adapter.safetensors
needle build --lora adapter.safetensors --layers 8 --platform linux-arm64 --out ./pi

The same ladder exists along width, where each sampled subnetwork is a depth and a channel prefix of every matrix, as in Matryoshka representations and MatFormer. The shipped Needle 3 exposes depth.

What it costs

At training time, a sampled step is cheaper than a full step, so the ladder adds no wall-clock time per step. The price is that 20% of steps update only part of the model, which the self-distillation term recovers.

At inference time there is no cost. A subnetwork of depth $d$ is exactly a $d$-layer model in operations, in cache and in file size. The blocks are the same bytes at every depth, so file size is a straight line in $d$.

A device can hold the full 29 MB file and choose a depth per request, a shallow pass for a simple command and the full stack for a hard one, with no second download. That part is not shipped yet.

References

1. He et al., Deep Residual Learning for Image Recognition, 2015.
2. Huang et al., Deep Networks with Stochastic Depth, 2016.
3. Fan et al., Reducing Transformer Depth on Demand with Structured Dropout, 2019.
4. Hinton et al., Distilling the Knowledge in a Neural Network, 2015.
5. Kusupati et al., Matryoshka Representation Learning, 2022.
6. Devvrit et al., MatFormer: Nested Transformer for Elastic Inference, 2023.
7. Cai et al., Once-for-All: Train One Network and Specialize it for Efficient Deployment, 2019.
8. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021.