AhaBulbStuff Everyone Should Know
AhaBulb: Stuff Everyone Should Know · ahabulb.com/ai-token-can-be-subword

Why is one written word not always one language-model token?

Start playing free

  1. A token can be a subword
  2. Every token is a paragraph
  3. Tokens must be whole sentences

🔒 The answer is for AhaBulb members. Members see the answer, why it's right and the official sources for every card of Stuff Every AI Expert Should Know, plus timed practice exams.

The first 50 cards of every test are free to play.

Play the free cards See membership

Like this card?
Play the game

References

  1. Google Machine Learning Glossary developers.google.com/machine-learning/glossary Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceHow does byte pair encoding tokenization handle a rare word the model has never seen?Artificial IntelligenceWhat does a learned embedding assign to each input token?Artificial IntelligenceWhich statement correctly distinguishes model parameters from tokens?

That's stuff everyone should know.