Nature's October 7, 2026 paper on byteification converts existing subword language models to read raw bytes using under 1% of pretraining compute. Here is how Bolmo works, what the benchmarks show, and where it falls short.
Ask a chatbot how many times the letter “r” appears in “strawberry” and you may get a confident wrong answer. The mistake is not carelessness. The model never saw the word as ten letters. It saw a handful of numbered chunks, and the letters inside those chunks were invisible to it.
On October 7, 2026, Nature published a paper that goes after the root of that problem. “Retrofitting language models to operate over bytes” describes byteification, a method for taking an already-trained language model and converting it to read raw bytes instead of tokens, using less than 1% of a typical pretraining budget. The open models built with it are called Bolmo, and the paper is worth understanding even if you never train a model yourself, because it changes what a tokenizer is allowed to be.
What a tokenizer does, and where it breaks
Large language models do not read text. They read integers. A tokenizer, usually built with byte-pair encoding or a close relative, chops text into pieces from a fixed vocabulary of tens of thousands of entries and maps each piece to an ID. Common words become single tokens. Rare words, names and code identifiers get split into fragments.
That design is efficient, since fewer tokens means a shorter sequence and cheaper attention. It also bakes in several quiet failures. Spelling, counting characters, reversing strings and handling unusual scripts all depend on information the model can only infer, because the tokenizer discarded it before the first layer ran. Andrej Karpathy’s long walkthrough of building a GPT tokenizer from scratch is still the clearest explanation of where these quirks come from.
A byte-level model sidesteps all of it. UTF-8 gives every string a representation built from just 256 possible values, so there is no vocabulary to outgrow and no out-of-vocabulary text. Here is what the same word looks like when you stop pretending it is a token:
text = "strawberry"
print(list(text.encode("utf-8")))
# [115, 116, 114, 97, 119, 98, 101, 114, 114, 121]
# ten bytes, ten letters, every "r" (114) visible to the modelWhy byte-level models stayed a research curiosity
The catch is length. A sentence that takes a dozen subword tokens can easily run several times that many bytes, and self-attention cost grows with the square of sequence length. Earlier byte-level architectures such as Meta’s Byte Latent Transformer (BLT), H-Net and Dynamic Token Pooling attacked this by grouping bytes into variable-length patches before the expensive layers. They worked, but they were trained from scratch. That meant paying for a full pretraining run just to get a model that, at the same size, usually trailed its tokenized cousins.
Byteification asks a different question: why not start from a strong subword model that already exists?
How byteification is built
The Bolmo team, with Benjamin Minixhofer as first author and co-authors including Luke Zettlemoyer, Noah A. Smith and Valentin Hofmann, keeps the original transformer as the core and wraps it in byte-aware machinery. Ai2’s description is that the approach “byteifies our open Olmo 3 models” rather than starting over. The pipeline has four parts.
| Component | Job |
|---|---|
| Local encoder | A light stack of mLSTM blocks embeds each byte and builds context-aware byte representations. |
| Boundary predictor | Decides where one patch ends and the next begins. It is non-causal and peeks one byte ahead, which makes it behave more like a real tokenizer than earlier causal designs. |
| Global transformer | The original Olmo 3 transformer processes the pooled patches, then the output is expanded back to byte resolution. |
| Local decoder | A second mLSTM stack refines the expanded states, and the output head predicts both the next byte and where the next boundary falls. |
The design is a translation layer. Bytes go in at the bottom, get compressed into patches the old transformer can digest, and get unpacked again on the way out. The heavy network in the middle is the expensive thing the team already owned.
Two training stages, 49.1 billion tokens
Stage one freezes the Olmo 3 transformer entirely. Only the encoder, decoder, boundary predictor and output head train, and their goal is imitation: behave like the tokenized model. This stage uses 9.8 billion tokens, roughly 43 billion bytes. The Nature paper reports that the boundary predictor passes 99% accuracy early in this stage, which suggests the patching problem is easier than it looked.
Stage two unfreezes everything. Another 39.3 billion tokens, about 173 billion bytes, lets the whole stack learn to exploit byte-level detail that the frozen version could not use. Total: 49.1 billion tokens, which the authors put at under 1% of a standard pretraining budget. Training data came from the Dolma 3 mix, open code datasets and a small set of character-level examples, per Ai2’s write-up. The write-up does not report GPU hours, so nobody outside the team can price the run from this material alone.
What the benchmarks show
Results depend on which comparison you care about, so here they are separated.
| Comparison | Reported outcome |
|---|---|
| Bolmo 7B vs. Olmo 3 7B (its own source) | Close on Ai2’s broad suite of math, STEM, QA, code and knowledge tasks |
| Bolmo 7B vs. Olmo 3 7B on character tasks | Large gains on the CUTE and EXECUTE benchmarks; character aggregate up by nearly 20 points |
| Bolmo 7B vs. BLT 7B (byte-level, trained from scratch) | +16.5% absolute on STEM tasks |
| Bolmo 7B vs. other 6.5B–7B byte-level models | Strongest overall across code, math, multiple-choice QA and character understanding; slightly behind TFree-Hat 7B on GenQA |
| Decoding speed at matched compression | About 125 bytes/s vs. about 150 bytes/s for the subword model |
The honest summary is that byteification roughly preserves general ability while buying a large improvement on anything that depends on characters, at a modest speed cost. That is a trade many applications would take. It is not a free lunch, and decoding that runs roughly 17% slower is the bill.
The weight-merging trick
The most practically interesting result involves instruction following. A freshly byteified base model scores 31.1% on IFEval, below the 35.4% of the original subword base. Retraining it with instruction data would cost real money.
Instead, the team computed the difference between the post-trained Olmo 3 weights and the Olmo 3 base weights, then added that difference to the byteified transformer. No further training. IFEval jumped to 67.4%, against 66.9% for the post-trained Olmo 3 checkpoint it was copying. The upshot is that a byteified model inherits the ecosystem of its parent: fine-tunes, safety tuning and instruction tuning made for the original can, in principle, be transplanted.
The authors applied the recipe to more than Olmo. Qwen3 8B and Llama 3 8B were converted too, producing models the Nature paper calls Bwen and Blama, with parameter counts shifting between roughly -0.7% and +4.5% from their sources.
Limits and open questions
- Speed. Decoding is slightly slower than the subword original at equal compression, though raising the average bytes per patch speeds it up.
- Merging is not guaranteed to travel. Ai2 notes that the weight-merging step relies on Olmo 3’s embeddings being resettable without loss, and that other model families may not behave the same way.
- Scale. Only 1B and 7B sizes are reported for Bolmo itself, with 8B conversions of other model families alongside. Whether the recipe holds at frontier size is untested in this work.
- Languages and domains. Multilingual and domain-specialized variants are listed as future work.
Why builders should care
Three groups stand to gain most. Code models suffer when identifiers and whitespace are split unpredictably. Genomic and protein models work on sequences where a “token” is an arbitrary imposition. Multilingual systems pay a hidden tax when a script is poorly covered by the vocabulary, since the same sentence can cost several times more tokens in one language than in another.
The weights and code are public. Ai2 has released Bolmo 7B and Bolmo 1B on Hugging Face, along with the bolmo-core training code and the bolmo_mix dataset. If you run evaluations on character-sensitive tasks, comparing Bolmo against its tokenized parent is a cheap afternoon experiment.
Frequently asked questions
Is this a new model or a new technique? Both. Byteification is the technique, and Bolmo is the family of models produced with it.
Does it fix the strawberry problem? It improves character-level reasoning substantially on the benchmarks the authors ran. That does not guarantee any single answer will be right, and nothing here tests that exact prompt.
Was this peer reviewed? The method appeared first as an arXiv preprint in December 2025, with a second version in February 2026. The Nature version was published October 7, 2026.
Sources
