Skip to content
yutils
Example

Input

Model: GPT-4o · Text: "Hello, nice to meet you."

Output

7 tokens · 24 characters · 0.01% of the 128K context · estimated input cost

Note

The same sentence tokenizes differently per model because vocabularies differ. GPT models are exact (tiktoken is public); Claude and Gemini are estimates because their tokenizers are not published.

Usage / FAQ

When to use

  • Check a prompt fits the model's context window before sending it
  • Size chunks by tokens when splitting a long document
  • Get a rough cost figure for an API call before budgeting
  • Compare how many tokens the same content costs in different languages
  • Decide how many documents fit in the context of a RAG pipeline

FAQ

Q.Why does the token count differ per model?
A.Because each tokenizer has its own vocabulary. Even within OpenAI, GPT-4 uses cl100k_base while GPT-4o uses o200k_base — and a larger vocabulary splits the same sentence into fewer tokens.
Q.What is the difference between "exact" and "estimate"?
A.OpenAI publishes tiktoken, so the browser can compute exactly what the API will charge. Claude and Gemini keep their tokenizers private, so the number shown is the GPT count with a correction factor applied — use each vendor's token-count API when you need the exact figure.
Q.The count doesn't match my actual bill.
A.The number here reflects the raw text you typed. A real API call adds message-format overhead (roles, system prompt, tool definitions), and output tokens are billed separately, so real usage runs higher.
Q.Do non-English languages cost more tokens?
A.Usually yes. Mainstream tokenizer vocabularies are trained mostly on English text, so an English word often maps to a single token while other scripts get split into several pieces. The same meaning then costs more money and more of the context window.
Fun facts

  • BPE (Byte Pair Encoding), the basis of today's LLM tokenizers, started life as a data compression algorithm in 1994. Sennrich et al. borrowed it in 2016 to handle rare and unseen words in machine translation, and it became the NLP default from there.

    arXiv — Neural Machine Translation of Rare Words with Subword Units

  • The same content costs wildly different token counts across languages. A 2023 study found gaps of up to 15× between languages in mainstream commercial tokenizers — and on APIs billed per token, that gap is a price difference.

    arXiv — Language Model Tokenizers Introduce Unfairness Between Languages

  • OpenAI's tiktoken ships a different vocabulary per model family — roughly 100k entries in cl100k_base (GPT-3.5/4) and about 200k in o200k_base (GPT-4o onward). A bigger vocabulary means fewer tokens for the same sentence, so switching models means redoing your prompt-length math.

    GitHub — openai/tiktoken

Related tools

See all in this category →