Technical
Token
The basic unit of text that LLMs process. Text is split into tokens (subwords) by a tokenizer, and the model predicts the next token in a sequence.
Explained at five levels
Level 1
A tiny piece of a word that the AI reads — like breaking "butterfly" into "butter" and "fly".
Level 2
The small chunks that AI breaks text into before reading it — usually parts of words. A sentence might be 10–20 tokens.
Level 3
The basic unit of text that LLMs process. Text is split into tokens (subwords) by a tokenizer, and the model predicts the next token in a sequence.
Level 4
A subword unit produced by a tokenizer (e.g., BPE or SentencePiece) that maps text to integer IDs consumed by the model. Context window size, cost, and latency all scale with token count.
Level 5
A discrete symbol from a finite vocabulary constructed via byte-pair encoding or unigram language modeling, serving as the atomic unit of the autoregressive factorization P(x₁,...,xₙ) = ∏P(xᵢ|x<ᵢ).
Definitions are educational summaries. Terminology can vary by source and context.
Sources
- NIST Trustworthy and Responsible AI Resource Center: Glossary — National Institute of Standards and Technology. Accessed 2026-07-20.