Transformer reasoning depth has a ceiling that is easy to miss. A language model can carry billions of parameters and dozens of layers and still lose the thread after following two or three references in a row. A preprint posted to arXiv on September 29, 2026, “Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It” by Zehao Jin, Ruixuan Deng and Junran Wang (arXiv 2609.36585), measures that ceiling and reports a surprisingly cheap way to raise it.

What follows covers the task the authors built, the intervention, the evidence for how it works, and the limits they state themselves. The paper is a preprint, and I found no peer-reviewed version. Every number below is the authors’ reported result, not something reproduced independently.

The task: following a chain of references

Each prompt in the study contains several reference chains. A chain begins with a root assignment that binds a single-token noun, and every later assignment names the variable before it. The model must output the root value of whichever chain the question asks about. Lines are arranged either grouped by depth or interleaved across chains. A toy version, written for this article and not taken from the paper, looks like this:

x1 = apple
y1 = pear
x2 = x1
y2 = y1
x3 = x2
question: what is x3?  answer: apple

Two terms carry the results. Chain length counts every assignment in a chain, root included, and a line is one assignment. Reach is the longest chain length the model answers with at least 80 percent accuracy. The task strips multi-hop lookup down to its plumbing, which is why a failure here points at something structural and not at a knowledge gap.

The baseline: three lines and out

Across the standard pretrained models, frozen reach sits between 1.4 and 3.6 lines, with a median of 2.2. The families tested include Qwen3 from 0.6B to 14B, Llama, OLMo-3 at 7B and 32B, and Gemma-3 from 4B to 27B, and the authors also report on DeepSeek-V4-Flash and on looped models called Ouro and Huginn. Looping the same layers again, the obvious way to add depth without adding parameters, buys very little in the pretrained versions.

Put plainly, a model with dozens of layers reliably resolves a chain of only two or three assignments. Something in the architecture or the training leaves most of the available depth unused for this job.

The intervention: one adapter at one layer

The fix is a rank-8 LoRA, a pair of small trainable matrices, applied to the residual stream at the input of a single layer. The update has the form h becomes s times h plus B A h, and only A, B and the scalar s are trained. Every weight in the base model stays frozen. In Qwen3-8B the adapter sits at layer 14 and adds 65,537 trainable parameters, about a thousandth of one percent of the model. In Ouro-1.4B it sits at layer 6 with 32,769 parameters and is applied in every loop.

Training combines cross-entropy on the answer with a KL penalty on WikiText-103, which is meant to keep the model’s ordinary text behavior from drifting. The standard runs take 1,200 AdamW steps on batches of 16 two-chain programs of up to 20 lines. A longer variant trains on programs of up to 40 lines.

What the numbers show

  • Qwen3-8B: exact accuracy on 24-line chains rises from 15.5 percent to 99 percent. The longer-trained adapter reaches about 50 lines.
  • Ouro-1.4B: reach grows to 60 lines after four loops and to at least 160 lines after eight.
  • Huginn: the longer-trained adapter reaches 58 lines after sixteen recurrences.
  • Placement matters. At layer 26 in the Qwen setup, all methods stay near chance.

The jump from about two lines to dozens is large enough that the authors’ framing, that depth goes unused until something activates it, is easy to follow. It is also a result on synthetic programs, a point the limits section returns to.

Wooden blocks falling like dominoes

How it seems to work: a relay through the middle layers

In frozen models, a line’s chain identity, meaning which chain it belongs to, is readable for only about three to five lines. With the adapter, a relay runs through the middle layers, roughly layers 16 to 22 in Qwen and 7 to 15 within each Ouro loop. Each line passes its chain identity along, and attention reaches progressively further up the chain, much like a bucket brigade that lengthens as it goes.

The mechanistic evidence comes from interventions. Frozen attention heads that normally read only the parent line take over the longer reads, and ablating those heads reduces reach. Blocking attention to the parent line stops the relay entirely. Blocking longer-distance attention shortens it, and in Ouro with three loops reach falls from 17.8 to 9.4. Probes also find ancestor names accumulating in the program lines, which fits the relay story.

Where to put the adapter

Placement decides everything, so the authors propose a measurement made on the frozen model alone. The cutoff layer is the first layer where restoring a pointer’s own token recovers less than half of its effect. For Qwen3-8B it falls at 20.5. The consequences of crossing it are steep: moving the adapter from layer 20 to layer 21 cuts reach from 20.5 lines to 5.2.

The authors preregistered a held-out test. The cutoff placed the limit within tolerance for three of four held-out models, and OLMo-3-32B was missed by five layers. Mean absolute error was 2.25 layers, against 3.45 for a simple rule that picks 45 percent of model depth. Across all nine models the cutoff’s error of 2.22 was about tied with a post-hoc rule of 48 percent depth, at 2.27. My reading is that a plain fraction of depth does nearly as well, so the practical edge of the measurement is modest. The authors themselves list placement precision as a limitation and say a local sweep is still needed.

Beyond synthetic chains: MuSiQue and invented facts

To test whether any of this transfers, the authors trained adapters on MuSiQue, a multi-hop question-answering set, and scored exact match on 900 development questions. At the earliest layers tested, gains were 11.4 points for Qwen3-8B, 9.4 for OLMo-3-7B and 17.9 for Llama-3.1-8B. In Qwen3-8B an adapter at layer 6 beat one at layer 30 by 13.0 points, and an early projection adapter kept most of the gain from adapting every layer. On fictional multi-hop facts, which rule out memorized knowledge, Qwen3-8B’s exact match rose from 41.2 percent to 97.4 percent.

Vintage telephone exchange relays

The authors are careful here. The MuSiQue results show that placement matters on real questions, but they do not directly confirm that the same relay mechanism is at work.

How to read the headline number

The 15.5-to-99 percent result is for 24-line chains, and the frozen median reach is 2.2 lines, so the evaluated chains are roughly eleven times longer than anything the untouched models handle reliably. The 15.5 percent starting point is also not zero, which says the frozen model still guesses right now and then, perhaps by picking up a nearby token. Several things are not visible in the figures I could access: how many test programs sat behind each percentage, how many random seeds were run, and how much the score varies between seeds. A reader weighing the claim should look for those details in the paper before leaning on a single figure.

The 99 percent figure should also be kept apart from the 50-line and 160-line results. They come from different models, different training lengths and, for Ouro, different numbers of loops, so they illustrate a pattern and do not form a single scaling curve.

A small experiment you could run

The reach metric is simple enough to rebuild for your own checkpoints. Generate programs in the format above with a known chain length, ask the model for the root value, and score exact matches over a few hundred programs at each length. Plot accuracy against length and mark where it first falls below 80 percent. That crossing point is your model’s reach, and it takes an afternoon to measure.

Repeat the measurement after attaching a small adapter at different layers, one position per run. If your results echo the paper’s, accuracy should hold at a placement in the middle of the network and collapse beyond a certain depth, and you will have a map of where your model can still be steered. If they do not echo it, you have learned something about the limits of the finding, which is just as useful.

Limits the authors state

  • The mechanistic tasks are synthetic, in-context programs.
  • Detailed traces cover mainly Qwen3-8B and Ouro-1.4B.
  • Adapters are trained separately for each task format.
  • Placement predictions have modest precision and need a local sweep.
  • The learned routing directions and the minimum sufficient rank are not identified.
  • The longest looped results use level-ordered chains and an adapter applied in every loop.

My own caveats are short. This is version 1 of a preprint. I read the paper’s text but did not run the code or inspect the interactive demo that it links. And the summary of the evaluation setup I could access did not say whether intermediate reasoning text was allowed, so check the paper’s evaluation section before comparing these numbers with chain-of-thought results.

Brass adapter ring beside a gearbox shaft

What this suggests for practitioners

This section is interpretation, not a finding from the paper. First, where an adapter attaches may matter as much as how large it is, and a layer sweep is cheap compared with training. Second, reach is a handy diagnostic: build a chain-following test from your own domain and find the number of hops at which accuracy drops below 80 percent. Third, keep the scope in view. A rank-8 adapter lifting synthetic chains from 15.5 to 99 percent says that depth can be unlocked, and it does not say that a general-purpose model got smarter.

Frequently asked questions

What is a LoRA? Low-rank adaptation: small trainable matrices added to a frozen model so that a few thousand to a few million parameters change instead of billions.

Does this make models better at everything? The reported evidence covers chain following and multi-hop question answering. The KL penalty on WikiText-103 is there to limit side effects, and the paper does not claim broad capability gains.

Can I try it? The paper links code and an interactive demo. Treat the headline numbers as unconfirmed until someone reproduces them.

Sources