Recursive Depth
Depth Without Parameters
A standard transformer passes a hidden state through each layer once, so depth and parameter count rise together. A looped transformer applies the same block repeatedly instead, giving an effective depth of layers multiplied by passes while the weights are stored once. The idea is older than the current interest in it, going back to Universal Transformers in 2018, and it has returned because test-time compute became something worth spending deliberately.
The Mechanism
Weight tying along the depth axis. What changes is not the arithmetic of a single block but how many times that arithmetic runs and what is paid for it.
| Property | Stacked layers | Looped block |
|---|---|---|
| Effective depth | Equal to the number of distinct layers. | Layers multiplied by passes, so 22 layers at two passes behaves like 44. |
| Stored weights | One set per layer. | One set for the whole loop, regardless of pass count. |
| Compute per token | Fixed by the architecture. | Scales with passes, and can differ per token. |
| Adding capability | Train a larger model. | Run more passes, including after training in some designs. |
| Memory during inference | One activation set and one KV cache. | Activation and cache pressure rise with each pass. |
Loop Control Strategies
Three ways to decide how many passes a token receives, in increasing order of how much machinery they require.
Every token goes through the same number of passes, chosen as a hyperparameter and unchanged at run time. Simple, predictable to serve, and wasteful on tokens that did not need the extra depth.
Cost: Nothing beyond the extra passes themselves.
The model learns when a token has been processed enough and can stop early, a form of adaptive halting. Compute concentrates on the tokens that are actually difficult.
Cost: A halting mechanism to train, and variable latency per sequence.
Lightweight routers assign each token its own recursion depth, and attention at each depth is computed over only the tokens still active there. Key and value pairs can be cached selectively or shared from the first pass.
Cost: Router training, plus the load-balancing problems that routing always brings.
Models Using It
| Model | Loop configuration | Effective depth | Control |
|---|---|---|---|
| Nanbeige4.2 3B parameters | Two fixed passes over a 22 layer stack | 44 block applications | Fixed |
| Ouro-Thinking 2.6B parameters | Up to four passes over 48 layers | Up to 192 applications | Learned exits |
| Mixture-of-Recursions 135M to 1.7B, research | Per-token recursion depth assigned by routers | Varies by token | Routed |
The pattern in the reported results is that a small model with loops competes with a substantially larger one without them, which is why the current interest is concentrated at the compact end rather than the frontier.
What It Costs
Parameter efficiency is not free efficiency. The savings are in storage and the costs are in memory traffic, training and serving predictability.
Activations and key-value cache grow with each pass, so the parameter saving is partly returned as memory pressure at inference. The model is smaller to store and not proportionally cheaper to serve.
Gradients flow through the same weights many times, which raises both compute and the difficulty of keeping training stable. Reported work tends to settle on few passes rather than many.
Adaptive halting and depth routing mean two requests of the same length can take different amounts of time, which complicates capacity planning and any hard latency budget.
More passes do not keep paying. Published configurations cluster at two to four, with the trade-off between quality and token efficiency turning over quickly beyond that.
Depth and Width
This is the depth-axis counterpart to mixture of experts. Both refuse to spend every parameter on every token; they disagree about which dimension to make conditional.
| Dimension | Mixture of experts | Recursive depth |
|---|---|---|
| What is conditional | Which parameters a token uses. | How many times a token is processed. |
| Parameters stored | Very large, most inactive per token. | Small, all active on every pass. |
| Scaling lever | Add experts. | Add passes. |
| Main serving cost | Memory to hold every expert. | Repeated compute and growing cache. |
| Routing decides | Which expert, at each layer. | Whether to continue, at each pass. |
They compose. A looped stack can contain expert layers, so a token can be routed to a subset of parameters and then sent through them more than once.
Related: Transformer Architecture for the block being reused, Mixture of Experts for the width-axis equivalent, LLM Performance for the inference costs loops add, and Fast and Slow Brain for spending variable compute per input at the system level rather than inside the model.