Nature describes byteification, which converts language models to bytes

On 7 October 2026, a paper on byteification was published in the journal Nature. It is a method that converts a ready-made language model to work with UTF-8 bytes instead of the usual tokens made of words and their fragments.

Most modern large language models split text into tokens before training, usually words or parts of words. This splitting hides information about individual characters, which gets in the way where meaning depends on a specific character: in computer code and biological sequences. Models that work with bytes do not have these limitations, but previously they were inferior to ordinary ones in quality.

How byteification works

The authors proposed a two-stage procedure for further training already ready open models. It took less than 1% of a typical pretraining budget, a total of 49.1 billion tokens.

Which models came out

This way Olmo 3 7B and OLMo 2 1B produced Bolmo 7B and Bolmo 1B, Qwen3 8B Base produced the Bwen 8B model, and Llama 3 8B produced the Blama 8B model. The byte versions surpassed earlier byte models of comparable size: Bolmo 7B gained 16.5 percentage points on STEM tasks against BLT 7B, trained from scratch. Bwen 8B statistically significantly outperformed all previous byte models and came close to the original Qwen in quality.

Byte models understand characters noticeably better than the originals. They can be sped up by increasing the average number of bytes per fragment, and ready further-trained versions of the original model are transferred to the byte one through task arithmetic without additional training. In quality, the byte versions are close to the original models, but they do not always surpass them.

Image: Large Language Models, Knowledge Graphs and Search Engines: A Crossroads for Answering Users' Questions, by Aidan Hogan, Xin Luna Dong, Denny Vrandečić, Gerhard Weikum, https://arxiv.org/abs/2501.06699v1 CC BY 4.0 · License

Translation editor: Салтанов А.