Do LLMs Have World Models? What Othello-GPT Shows and What It Doesn’t

Do LLMs have world models, or do they only juggle the statistics of the text they were trained on? Since 2022 the question has had an experimental form: train a sequence model on nothing but tokens, then look inside it for a variable that tracks the hidden state of whatever produced them. Find it, and you have evidence of a world model. Find only a lookalike, and the “surface statistics” camp gains ground.

People outside the field keep asking this too. Kenneth Li, lead author of the best-known study, framed his own popular write-up with that exact title: “Do Large Language Models learn world models or just surface statistics?” (Li 2023, The Gradient). The AI Guide newsletter ran a multi-part series called “LLMs and World Models”, and essays titled “Are LLMs Just Next Token Predictors?” turn up in ordinary web searches. What follows covers the probing evidence, the strongest critiques, and what a small experiment can add.

The question, stated precisely

“World model” is used loosely. Suppose a sequence comes from a process with hidden state: a board, a city map, a planet’s position and velocity. A model trained only on the outputs has a world model, in the narrow sense most of this literature uses, if three things hold. Its internal activations contain a variable that corresponds to the hidden state. That variable can be read out reliably by a simple decoder. And the model’s predictions actually depend on that variable, so editing it changes behavior the way the real state change would.

A stricter reading, argued by Vafa et al. (2024), adds a fourth condition: the representation should be coherent, meaning it treats two histories that lead to the same state as the same, and two histories that lead to different states as different. A model can predict the next token almost perfectly and still fail that test.

Why it matters: if next-token training reliably yields compact state representations, interpretability tools can audit what a model believes and robustness off-distribution becomes more plausible. If it yields a patchwork of heuristics, models will keep failing in ways that look baffling from outside. (For background on the training objective itself, see how large language models work.)

What Othello-GPT actually found

The founding study is Li et al. (2023), published at ICLR 2023. The authors trained an 8-layer GPT-style transformer to predict legal moves in Othello. Its 60 tokens named playable squares; it never saw a board. One version trained on 20 million synthetic games sampled uniformly from the game tree, another on far fewer championship games. The synthetic model’s top-1 prediction was a legal move in all but 0.01% of cases.

Then came the probes. A linear probe trying to read each square as black, white or empty from the activations did poorly, with error rates around 20 to 23% on the synthetic model against roughly 27 to 29% for an untrained network of the same shape. A two-layer MLP probe did far better, reaching 1.7% error at layer 7. The paper’s summary:

“we uncover evidence of an emergent nonlinear internal representation of the board state”

Li et al., 2023, abstract

The decisive piece was causal. Using gradient descent on the activations, the authors edited the probed board, then checked whether predicted legal moves matched the edited position. Across 1,000 natural test cases, the average number of wrong predictions fell from 2.68 (no edit) to 0.12. That is the third condition from the definition above, met at least in this setting.

A follow-up by Nanda, Lee and Wattenberg (2023, preprint) showed the nonlinearity was a framing problem. Probed as “mine, theirs or empty” relative to the player about to move, the board became linearly decodable, with accuracy reaching 99.6% at one layer, while the absolute black/white framing stalled near 75%. Adding a scaled probe direction to the residual stream, a single vector addition, matched or beat the earlier gradient-based edits. The representation was there; the original probe had been asking in the wrong coordinate system.

Beyond board games: maps, timelines and chess

Othello is synthetic and tiny. Two later lines of work pushed toward natural data. Gurnee and Tegmark (2023, preprint) probed Llama-2 models on place names and on dated people, artworks and headlines:

“We discover that LLMs learn linear representations of space and time across multiple scales.”

Gurnee and Tegmark, 2023, abstract

Their linear probes recovered latitude, longitude and dates from activations, and they reported individual neurons that track those coordinates. In chess, Karvonen (2024, preprint) trained a character-level model on real games and found board-state representations that could be edited, plus a latent estimate of player skill. Adding a derived skill vector raised the model’s win rate by up to 2.6 times.

Each finding shows that information about a hidden variable is present and, for Othello and chess, causally used. None shows that the whole representation hangs together.

The case against: coherence, inductive bias and the probe problem

The oldest objection is conceptual. Bender and Koller (2020) argued that form alone cannot carry meaning:

“In this position paper, we argue that a system trained only on form has a priori no way to learn meaning.”

Bender and Koller, 2020, abstract

Their argument concerns meaning, including reference to things outside the text. Othello-GPT does not refute it directly: a move sequence fully determines the board, so the state is a function of the form. Probing shows next-token training can recover such a function, not that it can handle worlds that text underdetermines.

The sharpest empirical critique is the coherence test of Vafa et al. (2024). They trained transformers on turn-by-turn directions from Manhattan taxi rides. On standard checks the models looked excellent, with generated routes valid 96 to 99% of the time. But when the authors reconstructed the street map each model implied, it contained impossible street orientations and flyovers. Under detours, the shortest-path model’s valid-route rate fell to 0.69 when only 1% of steps were random. Applying the same metrics to Othello, they found the championship-trained model failed badly (0.05 valid at 10% detours) while the synthetic one held at 0.97. Their verdict on such models:

“our evaluation metrics reveal their world models to be far less coherent than they appear”

Vafa et al., 2024, abstract

A second paper from overlapping authors, Vafa et al. (2025, preprint), introduced an “inductive bias probe.” Models trained on orbital trajectories predicted well but, adapted to new physics tasks, did not act as if they knew Newtonian mechanics: Kepler without Newton, in the authors’ framing.

Probing itself draws a methodological objection. Hewitt and Liang (2019) showed that a sufficiently expressive probe can score well by learning the task on its own, and proposed “control tasks” to measure probe selectivity. Belinkov (2022), reviewing the method in Computational Linguistics, stresses that probe accuracy shows correlation, not use. Information can be decodable from a representation the model ignores. Only interventions bridge that gap, and they have their own problems: an edit can push activations off the distribution the model was trained on. (A related debate about whether circuit-level explanations are faithful is covered in this piece on mechanistic interpretability under test.)

Three hypotheses, steelmanned

H1: Next-token prediction builds world models when they compress the data

The strongest version says that predicting sequences well forces a model to represent whatever variable makes the future conditionally independent of the past. In Othello, that variable is the board. For: linear decodability, causal edits, and a synthetic Othello model that passes even strict coherence metrics. Against: the same objective, on taxi routes and orbits, produced models that predict well yet are incoherent or fail to transfer.

H2: Models learn bundles of heuristics that resemble a world model on-distribution

Here the model stores many partial rules that cover the training distribution. Probes find state-like variables because heuristics correlate with state. For: detour fragility, incoherent maps, inductive-bias failures. Against: heuristics struggle to explain why editing one linear direction yields globally consistent Othello predictions.

H3: It depends on data coverage, and the question is badly posed

A middle view: the answer belongs to the training distribution, not to transformers or next-token loss. Random Othello games expose every state from many angles; championship games and shortest-path routes cover a narrow slice, which many incoherent maps fit equally well. Vafa et al. report exactly this pattern, with random-walk taxi data giving the most robust model (0.74 valid even at 75% detours). On this view, “do LLMs have world models?” should be replaced by “for which variables, under which data, and how coherent?”

H3 predicts that H1 and H2 will each be observed, in different regimes, which is roughly what the literature shows.

A small probing experiment built for this article

To make the probe-versus-model distinction concrete, a small simulation was built for this article: two tiny worlds with a known hidden state and a small recurrent network (48 tanh units, written in NumPy, trained by backpropagation through time) learning to predict the next token in each. It is a toy, not a transformer. Seeds are fixed; the script ran in about 160 seconds on two CPU cores.

• World A (linear world). An agent starts in the center of a 5×5 grid and moves up, down, left or right, choosing uniformly among moves that stay on the grid. Only the moves are visible. To predict the next move, the network must know where the agent is, because moves into a wall never occur.
• World B (nonlinear world). The agent starts on a random, unseen cell and picks any of the four moves; a move into a wall silently does nothing. After each move the network sees the color of the floor tile (three colors on a fixed random map). Predicting the color requires tracking position through both the unseen bumps and the color clues, so the best possible tracker is a Bayes filter, which the script computes exactly.

For each world, linear (multinomial logistic) probes were trained to read the agent’s cell from four kinds of features: the trained network’s hidden state; the same network before training; simple running counts of each token seen so far; and, in World A, a one-hot code of the last three tokens. Each probe had a shuffled-label control; probes were trained on 1,000 sequences and tested on 1,000 others.

# abridged from main.py and world.py; helper names stand in for inline code
G = 5; MOVES = np.array([[0,1],[0,-1],[-1,0],[1,0]])          # U D L R
COLOR = np.random.default_rng(42).integers(0, 3, (G, G))      # world B floor map

def sample_B(n, T, rng):        # unseen start, walls clamp silently, colour observed
    P = rng.integers(0, G, (n, 2)); X, Pos = [], []
    for t in range(T):
        m = rng.integers(0, 4, n); P = np.clip(P + MOVES[m], 0, G - 1)
        X += [m, 4 + COLOR[P[:, 0], P[:, 1]]]; Pos += [None, P.copy()]
    return np.stack(X, 1), Pos

def forward(th, X, V):          # Elman RNN, state t predicts token t
    inp = np.concatenate([np.full((len(X), 1), V), X[:, :-1]], 1)
    h, hs = np.zeros((len(X), H)), []
    for t in range(X.shape[1]):
        h = np.tanh(np.eye(V + 1)[inp[:, t]] @ th['W'] + h @ th['U'] + th['b'])
        hs.append(h)
    return np.stack(hs, 1)      # trained with Adam, 5,000-8,000 steps of 128 sequences

def probe(Ftr, ytr, Fte, yte, rng):  # linear probe + shuffled-label control
    lin  = LogisticRegression(max_iter=2000).fit(Ftr, ytr)
    shuf = LogisticRegression(max_iter=2000).fit(Ftr, rng.permutation(ytr))
    return lin.score(Fte, yte), shuf.score(Fte, yte)

# Bayes filter for world B: push belief through the (clamped) move,
# record argmax accuracy, then condition on the observed colour
for t in range(16):
    b = push_through_move(b, XB[:, 2*t])
    acc.append(argmax_hits(b, true_cell[:, t]))
    b *= (COLOR[None] == XB[:, 2*t+1] - 4); b /= b.sum((1, 2), keepdims=True)

# causal edit (world A): move the probe-read x-coordinate to the east wall
w = Ridge(1e-3).fit(h_train, x_train).coef_
h_edit = h + ((4 - probe_x(h)) / (w @ w))[:, None] * w
h_rand = h + random_direction_with_same_norm(h_edit - h)

Both networks learned their tasks. World A loss was 1.223 nats per move against a floor of 1.219, with 0.09% of probability on illegal moves. World B color loss was 0.196 nats, against 0.172 for the Bayes filter and 1.068 for guessing from color frequencies.

World B accuracy rises as color clues accumulate: over moves 9 to 16, the trained-network probe read the true cell 99.6% of the time, against 99.9% for the Bayes filter.

The causal test used 2,347 World A states in which the agent stood on the west wall (left moves illegal) with room to move vertically. Each hidden state was edited along the probe’s x-direction until the probe read “east wall,” and the script measured predicted left and right probabilities immediately and after one more vertical move through the recurrence. On a real east wall the correct answers are 1/3 for left and 0 for right.

What the toy experiment does and does not show

First, a perfect probe can be meaningless. In World A, position is literally a linear function of the input: x equals 2 plus the number of right moves minus the number of left moves. Running counts, with no network at all, are decoded with 100% accuracy. The trained network’s perfect score therefore means little on its own; the untrained network (30%) and last-three-tokens (20%) baselines are what give it meaning. This is the Hewitt and Liang point in miniature, and it applies to Othello, where the board is also a deterministic function of the move list.

Second, World B separates the cases. Once bumps are hidden and colors must be integrated, counts collapse to 23% while the trained network reaches 83%, within two points of the Bayes ceiling. The network computes something beyond a simple input summary, and it tracks the posterior over position closely. Still, the untrained network’s 35% shows that random recurrent features already carry real position information, so a probe score reported without that baseline would mislead.

Third, the edit was partly causal and messy. Moving the state along the probe direction cut the predicted chance of a now-“illegal” right move from 0.33 to 0.07, and after one more recurrent step the network’s chance of moving left rose from almost zero to 0.28. A random edit of identical size also lowered the immediate right-move probability (to 0.18), since large edits disturb a network non-specifically, yet it never raised the chance of moving left, and a step later its right-move probability climbed to 0.46. Still, the probe-edited network never fully adopted the new position. It sat between west and east, as expected if the probe direction only partly overlaps the one the network uses. The Othello vector edits of Nanda et al. were cleaner.

What it does not show: anything about transformers, natural language or scale. The worlds have 25 states, and neither coherence in the Vafa sense nor transfer was tested. A 48-unit bottleneck also pressures a network to compress, which may favor state-like representations more than a large model with room for many heuristics.

The author’s reading, and what would settle it

This section is opinion. The author’s reading of the evidence is that the yes-or-no question is settled in a narrow sense and wide open in the sense people care about. Narrowly, token-only training does produce causally used state variables; Othello-GPT and its linear follow-up establish that for one domain. Whether large language models trained on human text build coherent models of the domains they talk about is unsettled, and the strongest current evidence, from Vafa and colleagues, leans toward partial and fragile representations. H3, the data-coverage view, fits the full pattern best. It is also the most testable.

Studies that would move the question, each with a result that would count against a hypothesis:

• Coverage dose-response. Train identical models on one environment with data from narrow (shortest paths) to broad (random walks), holding token count fixed. H3 predicts coherence rises with coverage; a flat curve would count against it.
• Probe-versus-ceiling reporting. Where the Bayes-optimal state estimate is computable, report probe accuracy against that ceiling and against untrained-network and input-statistics baselines. Beating input statistics widely while sitting near the ceiling, as in World B, is far stronger evidence than raw accuracy.
• Edit-then-roll-forward tests. Edit the state, run several more tokens, and check whether predictions keep tracking the edited world. A real world model should carry the edit forward; a heuristic bundle should drift back.
• Coherence tests on natural-language models. Apply Myhill-Nerode-style tests to a general-purpose LLM in a domain where text fixes the state, such as chess notation. If general LLMs pass where specialist models fail, that supports H1 for broad pretraining.

A further prediction: models will look most like world-modelers where the hidden state is cheap to compute from text, and least where text underdetermines it. That would partly vindicate both Li et al. and Bender and Koller, which is more or less where Mitchell and Krakauer’s 2023 PNAS survey left the broader debate over understanding. (It also connects with the reasons why chatbots hallucinate: a model with an incoherent internal map will produce confident outputs that the map cannot support.)

Common questions

Does a high probe accuracy prove a model has a world model?

No. High accuracy shows information is decodable, not that the model uses it, and sometimes the information is trivially present in the input. Strong claims need baselines (untrained networks, input statistics), shuffled-label controls, and causal interventions.

Do chatbots like ChatGPT have world models?

Large language models linearly encode some world variables, such as coordinates and dates. Models trained on navigation and physics data, though, have built incoherent or non-transferable internal maps. Whether general chatbots have coherent world models of any particular domain has not been established.

Why does the training data matter so much?

Narrow data, such as expert games or shortest routes, leaves many internal maps equally consistent with what the model sees, so nothing forces the true one. Broad data, such as random play, exposes more states and transitions, which is where the most coherent models in the literature came from.

Further reading

• Artificial Intelligence: A Guide for Thinking Humans by Melanie Mitchell. A readable book-length treatment of what AI systems do and do not understand, by a co-author of the PNAS survey cited here. See it on Amazon (affiliate link)

As an Amazon Associate I earn from qualifying purchases. A commission may be earned if you buy through this link, at no extra cost to you.

References

1. Belinkov, Y. (2022). Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1), 207–219. MIT Press; preprint arXiv:2102.12452.
2. Bender, E. M., & Koller, A. (2020). Climbing towards NLU: On meaning, form, and understanding in the age of data. Proceedings of ACL 2020, 5185–5198. doi:10.18653/v1/2020.acl-main.463.
3. Gurnee, W., & Tegmark, M. (2023). Language models represent space and time. Preprint. arXiv:2310.02207.
4. Hewitt, J., & Liang, P. (2019). Designing and interpreting probes with control tasks. Proceedings of EMNLP-IJCNLP 2019. arXiv:1909.03368.
5. Karvonen, A. (2024). Emergent world models and latent variable estimation in chess-playing language models. Preprint. arXiv:2403.15498.
6. Li, K. (2023). Do large language models learn world models or just surface statistics? The Gradient. thegradient.pub/othello.
7. Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2023). Emergent world representations: Exploring a sequence model trained on a synthetic task. ICLR 2023. arXiv:2210.13382; OpenReview.
8. Mitchell, M., & Krakauer, D. C. (2023). The debate over understanding in AI’s large language models. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.2215907120.
9. Nanda, N., Lee, A., & Wattenberg, M. (2023). Emergent linear representations in world models of self-supervised sequence models. Preprint. arXiv:2309.00941.
10. Vafa, K., Chen, J. Y., Rambachan, A., Kleinberg, J., & Mullainathan, S. (2024). Evaluating the world model implicit in a generative model. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). NeurIPS proceedings; arXiv:2406.03689.
11. Vafa, K., Chang, P. G., Rambachan, A., & Mullainathan, S. (2025). What has a foundation model found? Using inductive bias to probe for world models. Preprint. arXiv:2507.06952.