The Reversal Curse: Why LLMs Learn ‘A Is B’ but Not ‘B Is A’

The reversal curse is the name researchers gave to an odd gap in how language models store facts: train a model on “Tom Cruise’s mother is Mary Lee Pfeiffer” and it can name the mother, yet ask “Who is Mary Lee Pfeiffer’s son?” and it often draws a blank. A person who learned the first sentence would often manage the second. For a model, the two questions turn out to be different skills.

The open question is not whether the gap exists. Several groups have reproduced it. What remains unsettled is why it exists and what kind of fix would remove it at the root rather than paper over it. Is it a side effect of predicting text one token at a time, a quirk of how gradient descent shapes weights, a deeper failure to bind concepts into stable representations, or simply a shortage of data written in both orders? Each implies a different repair.

People outside the field ask versions of this too. When the original paper was cross-posted to LessWrong, the thread ran to 74 comments. Readers asked why nobody simply trains on reversed text, whether “is” even licenses the reverse inference in plain English, and whether humans show the same lopsidedness when recalling names. Those three questions frame most of what follows.

What the reversal curse is, precisely

The term comes from Berglund and colleagues (2023), later presented at ICLR 2024. Their definition is short:

“If a model is trained on a sentence of the form “A is B”, it will not automatically generalize to the reverse direction “B is A”. This is the Reversal Curse.”

Berglund et al., 2023, arXiv:2309.12288

Two features of that definition matter. First, it concerns facts learned during training, stored in the weights. Second, “automatically” carries the weight: the claim is that seeing one direction confers no measurable benefit on the other, not merely a reduced one.

Their cleanest test used invented people, so no prior knowledge could leak in. GPT-3 models from 350 million to 175 billion parameters were fine-tuned on 30 fictitious facts such as “Daphne Barrington is the director of ‘A Journey Through Time'”, each paraphrased 30 times. For the largest model, facts learned in the description-to-name order were recalled 96.7% of the time in that same order and 0.1% of the time in reverse. More telling, the log-probability the model assigned to the correct name in the reverse query was no higher than the log-probability it gave a random name. Statistical tests found no difference. Hyperparameter sweeps, a 40,000-document version and prompt tuning did not lift reverse accuracy.

A second experiment used real knowledge. From the 1,000 most popular celebrities on IMDb, the authors collected 1,573 child-parent pairs. GPT-4 named the parent given the child 79% of the time, but named the child given the parent only 33% of the time, even when allowed ten attempts per question.

Evidence that it is a training effect, not a prompting trick

One obvious objection is that the model “knows” the reverse fact but the prompt fails to elicit it. Three independent lines of evidence argue against that reading.

The first is from controlled pretraining. Allen-Zhu and Li (2023), in a preprint from their “Physics of Language Models” series, trained models from scratch on synthetic biographies and then asked them to search knowledge backwards, from an attribute to the person who has it. Their abstract is blunt:

“Moreover, their performance in inverse knowledge search is virtually 0%, regardless of the prompts.”

Allen-Zhu and Li, 2023, arXiv:2309.14402 (preprint)

The second comes from attribution. Grosse et al. (2023), also a preprint, used influence functions to estimate which training sequences most affect a given output of large pretrained models. They reported that the influence of a relevant training sequence on a completion largely vanishes once its key phrases appear in the other order:

“influences decay to near-zero when the order of key phrases is flipped”

Grosse et al., 2023, arXiv:2308.03296 (preprint)

The third is the in-context contrast. Berglund and colleagues found that when “A is B” is placed in the prompt rather than in the training data, models infer “B is A” without trouble; GPT-3 scored between 96.67% and 100% in both directions. So the model is not incapable of reversal as an operation. It fails to perform reversal on knowledge it absorbed through gradient updates. That distinction, between reasoning over text in the context window and retrieving facts stored in parameters, is the hinge of the whole debate. (For readers who want the basic machinery first, x9m8 has a plain explainer on how large language models work.)

Do people have a reversal curse?

Psychology has argued about a close cousin of this problem for decades. In paired-associate learning, subjects study word pairs and are then cued with one word to recall the other. The associative symmetry hypothesis holds that learning A-B forms a single bidirectional link. Kahana (2002) tested this by correlating recall across successive forward and backward tests and found near-perfect correlations, which supported symmetry for unrelated word pairs.

That picture got more complicated. Popov et al. (2019), writing in Memory & Cognition, found that for semantically related pairs, practice in one direction did not carry over to the other, a pattern they read as two separate directional associations. Symmetry, on this evidence, depends on what the learner already knows. So the honest summary is that humans show some reversal asymmetry, under some conditions, but nothing like the near-total failure seen in fine-tuned models on invented facts.

Four competing explanations

The hypotheses below overlap, but separating them clarifies what each would predict.

1. The training objective: left-to-right prediction

The most direct account blames next-token prediction. If every training example asks the model to predict B given A, nothing in the loss rewards predicting A given B. Lv et al. (2023) argued in a preprint that “the reversal curse can stem from the specific training objectives.” Kitouni et al. (2024), another preprint, generalized the point as a “factorization curse”: a model trained under one ordering of a sequence’s joint probability learns only that factorization. They report that neither scale, nor reversed tokens, nor naive bidirectional training reliably fixes retrieval, while objectives that are agnostic to factorization order help substantially.

Evidence for: the in-context contrast, the influence-function result, and the success of order-agnostic objectives. Evidence against, or at least complication: the claim that naive bidirectional training fails suggests the objective alone is not the whole story.

2. Weight asymmetry from training dynamics

A more mechanistic version comes from Zhu et al. (2024), a preprint that analyzed gradient dynamics in bilinear models and one-layer transformers. Their conclusion, paraphrased: raising the effective weight from token A to token B during training does not raise the weight from B to A, and this follows from the loss function together with the parameter space being optimized. The curse is then a predictable consequence of how updates flow, not a mystery. The limitation is generality. Results proved for one-layer models may not survive the many-layer, multi-token setting where real facts live.

3. A binding problem in representations

Wang and Sun, in work presented at ICLR 2026, propose that

“the Reversal Curse in LLMs is a manifestation of the long-standing binding problem”

Wang and Sun, 2025, arXiv:2504.01928

Their argument is that transformers form inconsistent and entangled representations of the same concept depending on where it appears, so the “Mary Lee Pfeiffer” that the model reaches when reading forward is not cleanly the same object it would need to start from when reading backward. They report that a joint-embedding predictive architecture, plus memory layers that keep concept representations separate, breaks the curse without special data augmentation or non-causal masking. If that holds up, the problem is less about the objective and more about where the model keeps its concepts.

4. A data problem dressed up as an architecture problem

The deflationary view says models learn whatever directions their data contain, and real text rarely states most facts both ways. Golovneva et al. (2024), in a preprint, argued that because word frequencies follow Zipf’s law, the curse persists even at very large data scales: many facts appear in one direction only. Their remedy, reverse training, trains on reversed strings while keeping entity names in their original order. They report better reversal performance at equal compute and no loss on standard benchmarks at equal data. On this account the curse is real but shallow, a gap in coverage rather than a flaw in kind.

The weakness of the purely data-based story is that it predicts any symmetric augmentation should suffice, and the factorization-curse results suggest naive reversal does not always work. That tension is unresolved.

A small experiment built for this article

To see the mechanism with nothing hidden, a small simulation was built for this article in Python and NumPy. It trains a tiny next-token model on two-token prompts and asks it to predict a third token. There are 200 “name” tokens and 200 “description” tokens, paired one to one. Every pair is trained as name IS description. For half the pairs, called auxiliary pairs, the model also sees the reverse form description IS-OF name. The other 100 pairs, the test pairs, are seen only forward. The question is whether the model can answer description IS-OF ? for test pairs, ranking the correct name above the other 99 test names. Chance is 1%.

The model is one hidden layer of 128 tanh units over 32-dimensional embeddings, trained with full-batch Adam for 1,500 steps, five random seeds per condition. Two design choices vary. The output layer is either untied (a separate weight matrix) or tied (it reuses the input embedding table, as GPT-2 did). And the training set either stays as is or adds token-reversed copies of every example, a crude stand-in for reverse training. Backpropagation does all the learning; our plain guide to backpropagation covers the basics. An abridged listing:

# Abridged from rc.py (full script: one hidden layer, Adam, 1,500 steps, 5 seeds)
import numpy as np
N = 100                                   # 100 test pairs + 100 auxiliary pairs
NAMES, DESCS = np.arange(2*N), np.arange(2*N, 4*N)
IS, ISOF, SI, FOSI = 4*N, 4*N+1, 4*N+2, 4*N+3   # relation tokens

def data(reverse_train):
    s  = [(NAMES[i], IS, DESCS[i]) for i in range(2*N)]        # "A is B", all pairs
    s += [(DESCS[i], ISOF, NAMES[i]) for i in range(N, 2*N)]   # "B is-of A", aux only
    if reverse_train:                                          # token-reversed copies
        s += [(z, {IS: SI, ISOF: FOSI}[r], x) for x, r, z in list(s)]
    return np.array(s)

def logits(P, ctx, tied):
    x = np.concatenate([P['E'][ctx[:, 0]], P['E'][ctx[:, 1]]], 1)
    h = np.tanh(x @ P['W1'] + P['b1'])
    # tied: the output layer reuses the input embedding table E
    return (h @ P['P']) @ P['E'].T if tied else h @ P['W2']

# Training: cross-entropy on the third token only (next-token loss),
# full-batch Adam, lr 0.01, L2 1e-4.  Then the key test:
def reverse_acc(P, tied):
    i = np.arange(N)                                     # test pairs only
    ctx = np.stack([DESCS[i], np.full(N, ISOF)], 1)      # "B is-of ?"
    lg = logits(P, ctx, tied)[:, NAMES[i]]               # rank the 100 test names
    return (lg.argmax(1) == i).mean()                    # chance = 0.01

Results from that run (mean over five seeds; “gap” is the log-probability of the correct name minus the mean log-probability of the other test names, in nats):

The untied model reproduces the curse exactly. It learns every forward fact, learns the reverse form for the 100 pairs that showed it, and still sits at chance on the test pairs. Its gap of 0.00 nats echoes the Berglund finding that the correct name scores no better than a random one. The reason is visible in the code: a test name never appears as a target, so its output weights are only ever pushed down, and the input embedding of a test description is never trained at all, because test descriptions never appear as inputs. Nothing connects them. Seeing 100 examples of the inverse relation teaches the model the relation token, not the missing links.

The tied model was the surprise. Without any reversed data it answered 99.4% of reverse queries correctly. But two control probes show what it actually learned. Given a test description followed by the forward token IS, which in this data never precedes a name, it still returned the paired name 99.4% of the time. Given the reversed-format token, which it had never been trained on as an input, it did so 99.8% of the time. Tying lets a description’s embedding be pulled toward its partner during forward training, so the model ends up retrieving “whatever is linked to this token” regardless of the relation asked. That is symmetric association, close to what Kahana described for unrelated word pairs, not knowledge of an inverse relation.

What the toy shows, and what it does not

It shows one concrete route to the curse: when the representation a token receives as an output is disconnected from the representation it receives as an input, forward training leaves nothing for a reverse query to use. That fits the weight-asymmetry account of Zhu et al. and the objective-based account of Lv et al. It also shows the cheap fix working: once reversed copies are in the training set, reverse accuracy is perfect in both architectures.

It does not show that tied embeddings would cure large models. Real entities span several tokens, facts are stored in deep MLP and attention layers rather than in a single embedding lookup, and nothing in the published results suggests that this architectural detail rescues large models. The tied result is better read as a warning about toy tests: a model can pass a reversal benchmark by becoming blind to relational direction, which would be a different failure. Any serious reversal benchmark needs the kind of control probe used here, checking that the model can tell “A’s son” from “A’s mother” rather than simply returning the most associated name. Nor does a five-seed run on 400 tokens say anything about scale.

There is a related limit on how far trained models follow chains of references; a separate x9m8 piece on why transformers stop following references early looks at that from the depth side.

The author’s reading, and tests that would settle it

What follows is opinion, labeled as such. My reading is that the evidence favors a layered answer. At the bottom, the next-token objective and the gradient dynamics it produces guarantee that a fact learned one way creates no reverse pathway unless something else builds it. That much looks close to settled for small models and is consistent with every large-model observation. Whether that something else must be data (reverse training), a different objective (order-agnostic prediction) or a different architecture (better concept binding) is the genuinely open part. I lean toward thinking the binding account will matter most for multi-hop reasoning, while data augmentation will remain the practical fix for plain fact retrieval, but I hold that loosely.

Specific predictions and studies that would move the question:

• A clean real-model test. Fine-tune an open model on invented facts, as Berglund did, but score with three probes: the reverse query, a wrong-relation query (ask for the child when trained on the parent) and an unrelated-relation query. A true fix raises only the first. If all three rise together, the fix is association, not reversal.
• Multi-token entities in the toy. Extend this simulation so names span two or three tokens. Prediction: the tied-embedding shortcut disappears, because no single embedding carries the whole entity.
• Objective versus architecture, head to head. Train matched models with an order-agnostic objective and with a binding-oriented architecture on identical synthetic biographies. If the binding hypothesis is right, the architectural fix should transfer better to two-hop questions that neither model saw.
• A dose-response curve for data. Vary the share of facts stated in both directions from 0% to 100%. The data hypothesis predicts reverse accuracy tracks that share smoothly; the factorization hypothesis predicts it plateaus below full recovery under plain next-token training.
• A human comparison with matched materials. Teach people invented name-description pairs in one direction and test reverse recall, using the alternating-direction design from Popov et al. That would tell us how much of the gap is peculiar to machines.

Common questions

Is the reversal curse just a bug in older models?

No published evidence shows it simply vanishes with size. Berglund et al. found it across GPT-3 sizes up to 175 billion parameters and in GPT-4 on real celebrities, and later work treats it as a property of the training setup. Newer models may answer famous reverse questions because their training text contains both directions.

Why can a chatbot reverse a fact I type into the prompt?

Facts in the context window are processed by attention, which can look at every token in either direction. Facts stored in weights were written there by gradient updates during training, and those updates were directional. The curse applies to the second kind.

Does training on reversed text fix it?

Partly. Golovneva et al. report large gains from reverse training that keeps entity names intact, and the toy in this article went from chance to perfect accuracy with reversed copies. Kitouni et al. caution that naive reversal does not reliably fix retrieval in general, which is why the question is still open.

Is “B is A” always a valid inference?

Not in ordinary English. “A dog is a mammal” does not reverse. The research uses relations that do invert, such as name and unique description or parent and child, so the failure is not about logic but about retrieval.

References

1. Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., & Evans, O. (2023). The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”. arXiv:2309.12288; presented at ICLR 2024. https://arxiv.org/abs/2309.12288; ICLR 2024 record
2. Allen-Zhu, Z., & Li, Y. (2023). Physics of Language Models: Part 3.2, Knowledge Manipulation. arXiv:2309.14402 (preprint). https://arxiv.org/abs/2309.14402
3. Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., Hubinger, E., Lukošiūtė, K., Nguyen, K., Joseph, N., McCandlish, S., Kaplan, J., & Bowman, S. R. (2023). Studying Large Language Model Generalization with Influence Functions. arXiv:2308.03296 (preprint). https://arxiv.org/abs/2308.03296
4. Kahana, M. J. (2002). Associative symmetry and memory theory. Memory & Cognition, 30, 823–840. https://doi.org/10.3758/BF03195769
5. Popov, V., Zhang, Q., Koch, G. E., Calloway, R. C., & Coutanche, M. N. (2019). Semantic knowledge influences whether novel episodic associations are represented symmetrically or asymmetrically. Memory & Cognition, 47, 1567–1581. https://doi.org/10.3758/s13421-019-00950-4
6. Lv, A., Zhang, K., Xie, S., Tu, Q., Chen, Y., Wen, J.-R., & Yan, R. (2023). An Analysis and Mitigation of the Reversal Curse. arXiv:2311.07468 (preprint). https://arxiv.org/abs/2311.07468
7. Kitouni, O., Nolte, N., Bouchacourt, D., Williams, A., Rabbat, M., & Ibrahim, M. (2024). The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More. arXiv:2406.05183 (preprint). https://arxiv.org/abs/2406.05183
8. Zhu, H., Huang, B., Zhang, S., Jordan, M., Jiao, J., Tian, Y., & Russell, S. (2024). Towards a Theoretical Understanding of the ‘Reversal Curse’ via Training Dynamics. arXiv:2405.04669 (preprint). https://arxiv.org/abs/2405.04669
9. Wang, B., & Sun, H. (2025). Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure. arXiv:2504.01928; presented at ICLR 2026. https://arxiv.org/abs/2504.01928; ICLR 2026 record
10. Golovneva, O., Allen-Zhu, Z., Weston, J., & Sukhbaatar, S. (2024). Reverse Training to Nurse the Reversal Curse. arXiv:2403.13799 (preprint). https://arxiv.org/abs/2403.13799
11. Paper: LLMs trained on “A is B” fail to learn “B is A” (2023). Cross-post of Berglund et al. with reader discussion. LessWrong. https://www.lesswrong.com/posts/SCqDipWAhZ49JNdmL/paper-llms-trained-on-a-is-b-fail-to-learn-b-is-a