byte-pair encoding (BPE)

A subword tokenization method that starts from individual characters and repeatedly merges the most frequent adjacent pair.

Byte-pair encoding (BPE) builds a subword vocabulary by starting from individual characters (or bytes) and iteratively merging the most frequent adjacent pair of symbols into a new symbol, until a target vocabulary size is reached. Applying the learned merge sequence to new text is how BPE tokenizes it.

Examples

If "th" is the most frequent adjacent pair in a training corpus, BPE's first merge creates a single "th" symbol.