AhaBulbStuff Everyone Should Know
AhaBulb: Stuff Everyone Should Know · ahabulb.com/ai-bpe-subword-tokens

How does byte pair encoding tokenization handle a rare word the model has never seen?

Start playing free

  1. It replaces the word with an unknown marker
  2. It splits it into smaller subword units
  3. It looks the word up in a dictionary

🔒 The answer is for AhaBulb members. Members see the answer, why it's right and the official sources for every card of Stuff Every AI Expert Should Know, plus timed practice exams.

The first 50 cards of every test are free to play.

Play the free cards See membership

Like this card?
Play the game

References

  1. Sennrich et al., Neural Machine Translation of Rare Words with Subword Units (arXiv 1508.07909) arxiv.org/abs/1508.07909 Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhy is one written word not always one language-model token?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?Artificial IntelligenceWhat did Alan Turing call his 1950 test of machine intelligence?

That's stuff everyone should know.