Text, tokens, and IDs
Separate text pieces from their numeric identifiers.
- Separate text pieces from their numeric identifiers.
A tokenizer converts text into pieces called tokens and maps them to IDs in a vocabulary. Tokens may be words, word pieces, characters, or bytes; spaces and punctuation also matter. The same text can have different token counts with different tokenizers.
A closer look
Our toy lab uses a simple word tokenizer, not a production LLM tokenizer. An ID is just a label: ID 8 is not twice the meaning of ID 4. A context window limits how many tokens a model can use; the application may select, summarize, or discard material to fit.
Explain it yourself
If two tokens have adjacent IDs, must their meanings be similar?
Compare with an explanation
No. IDs identify vocabulary entries; their numerical distance does not measure meaning.
You mark this yourself. Nothing is graded here.