Tokens and Context Windows Explained: Why They Set Your API Bill

A token is the smallest unit of text that an LLM reads. It is not always a whole word. Common words like "the," "is," and "for" are single tokens. Longer or less common words get split into pieces:

• "Hello" = 1 token
• "embedding" = 1 token (common in AI contexts)
• "Nairobi" = 2 tokens (roughly "Nai" + "robi")
• "authentication" = 2 to 3 tokens
• "KES 2,500" = 4 to 5 tokens (numbers and punctuation split aggressively)

The exact split depends on the model's tokenizer. OpenAI models use a tokenizer called tiktoken. Anthropic's Claude uses its own. You can check token counts before sending a request using these libraries.

// Count tokens before sending (OpenAI example)
import { encoding_for_model } from 'tiktoken';

const enc = encoding_for_model('gpt-4o');
const tokens = enc.encode('M-Pesa is a mobile money service in Kenya');
console.log(tokens.length); // ~10 tokens
enc.free();

Rule of thumb for English: 1 token is roughly 0.75 words, or 4 characters. A 1,000-word article is about 1,300 to 1,400 tokens.

The context window is the total number of tokens the model can process in one request. This includes everything: the system prompt, conversation history, retrieved documents (for RAG), the user's message, and the model's response.

Think of it as the model's working memory. Anything outside the window does not exist for the model. If your conversation history grows beyond the window, the oldest messages get dropped (or you have to drop them yourself).

Context window sizes vary by model. As of mid-2026, windows range from 8,000 tokens on small models to over 1,000,000 on the largest. Bigger windows let you process more data per request, but they also cost more because you are charged for every token in the window.

A practical breakdown of what fits in different window sizes:

• 8K tokens: A few pages of text. Enough for a short conversation or a single document section.
• 32K tokens: A medium-length report or a conversation with 20 to 30 turns of history.
• 128K tokens: A full novel or a large codebase file. Enough for most RAG applications.
• 1M tokens: Multiple books or entire codebases. Useful for analysis tasks but expensive per request.

LLM APIs charge separately for input tokens (what you send) and output tokens (what the model generates). Output tokens are typically 2x to 4x more expensive per token than input tokens.

A single API call's cost is:

cost = (input_tokens * input_price_per_token)
     + (output_tokens * output_price_per_token)

This means two things for your architecture:

• Long system prompts are an ongoing cost. If your system prompt is 500 tokens, you pay for those 500 tokens on every single request. Over thousands of requests per day, that adds up.
• Conversation history grows linearly. Each new message in a conversation adds tokens. A 20-turn conversation might have 3,000 to 5,000 tokens of history, all of which you send (and pay for) with every new message.

Cost optimization strategies:

• Keep system prompts concise. Every word costs money at scale.
• Summarize old conversation history instead of sending the full transcript.
• For RAG, retrieve fewer, more relevant chunks rather than dumping everything in.
• Set a max_tokens limit on responses to prevent the model from generating unnecessarily long outputs.

In a chatbot, the conversation eventually exceeds the context window. You need a strategy:

Strategy 1: Sliding window. Keep the system prompt plus the last N messages. Drop the oldest messages when the total exceeds a threshold.

function trimConversation(
  messages: Message[],
  maxTokens: number,
  systemPrompt: Message
): Message[] {
  const systemTokens = countTokens(systemPrompt.content);
  let totalTokens = systemTokens;
  const trimmed: Message[] = [];

  // Walk backwards from the most recent message
  for (let i = messages.length - 1; i >= 0; i--) {
    const msgTokens = countTokens(messages[i].content);
    if (totalTokens + msgTokens > maxTokens) break;
    totalTokens += msgTokens;
    trimmed.unshift(messages[i]);
  }

  return [systemPrompt, ...trimmed];
}

Strategy 2: Summarize and compress. Periodically ask the model to summarize the conversation so far, then replace the old messages with the summary. This preserves context while reducing token count.

Strategy 3: Use a larger model for long contexts. If you genuinely need the full history (legal analysis, code review), pick a model with a larger context window and accept the higher per-request cost.

Tokenizers are trained primarily on English text. Other languages, especially those with non-Latin scripts, tend to use more tokens per word. Swahili, which uses Latin script, is relatively efficient but still uses more tokens than English for the same meaning.

For example, "Karibu Kenya" (Welcome to Kenya) might use 3 to 4 tokens, while "Welcome to Kenya" uses 3. The difference is small for Swahili, but languages like Chinese, Arabic, or Hindi can use 2x to 3x more tokens per word.

This matters if you are building a multilingual app. Your costs and context window usage will vary by language. Test with representative text in each language you support to get accurate estimates.