WeeBytes
What are tokens?
AI & MLLearn
Advanced

What are tokens?

AI doesn't read words like we do. It breaks text into tokens — chunks that might be whole words, parts of words, or punctuation. 'ChatGPT' might be 2-3 tokens. This matters because: 1) You're charged per token on paid APIs, 2) There's a maximum tokens the AI can process at once (context window), 3) Longer outputs cost more. Understanding tokens helps you use AI more efficiently.

token
Swipe
What are tokens? | WeeBytes