Vietnamese GPT-2, pretrained from scratch
A Vietnamese tokenizer and GPT-2 trained from random initialization on 2.64B tokens.
Overview
GPT-2's English vocabulary shreds Vietnamese: every accented letter costs two or three byte-level tokens. We trained our own byte-level BPE tokenizer with the same 50,257-token budget, spent on Vietnamese.
We curated and deduplicated a mix of news and Wikipedia into roughly 2.64B training tokens, pretrained GPT-2 from random initialization, then continued pretraining on a corpus of five-word quatrains as a controlled study of how far continued pretraining shifts style.
The model ships as a Docker trainer with a FastAPI backend and a Next.js chat interface.
On sample Vietnamese sentences it needs a quarter to a third of the tokens GPT-2 does, so the same 1,024-token context holds roughly three times as much text.
Compare tokenizers
tokens with GPT-2
tokens with our tokenizer