Tokenization is the process of breaking text into smaller units called tokens before it is processed by a language model. These tokens become the input used for AI inference and response generation.
Voice AI systems tokenize user queries, conversation history, prompts, and retrieved knowledge before sending them to large language models for processing and response generation.