AI Glossary · Definition
What is Token?
Token: A token is a small unit of text, often a word or part of a word, that AI models read and write; usage limits and costs are usually measured in tokens.
By DAIDU EditorialUpdated
Token, explained
Language models do not process text letter by letter or word by word. They split it into tokens. A common English word may be one token, while a long or rare word may be split into several, and the same sentence can use a different number of tokens in Arabic than in English. Tokens matter in business for two reasons: API pricing is usually per token, and each model has a maximum number of tokens it can handle at once, called its context window.
Example
A team estimates the cost of summarising 5,000 support emails by counting the tokens in a sample and multiplying by the provider's price per token.