If you’re running AI workflows at scale—content generation, summarization, classification—you’ve probably noticed your API bills don’t match your mental math. You estimate 5,000 tokens per request, the platform charges you for 7,200, and there’s no line item explaining the gap.
The problem isn’t hidden fees. It’s that token counting isn’t standardized, and most platforms count more than the text you send.
What platforms actually count as tokens
When you send a prompt to an AI model, you’re not just billed for the words in your input and output. You’re also charged for:
- System instructions — the hidden prompt that sets behavior (e.g., “You are a helpful assistant”). On some platforms, this adds 50–150 tokens per request, whether you write it or use the default.
- Formatting tokens — JSON structure, markdown syntax, and special delimiters all count. A 200-word response formatted as JSON can cost 20–30% more than plain text.
- Function call schemas — if you’re using tools or structured outputs, the schema definition gets tokenized and billed every single request, even if the model doesn’t invoke the function.
- Conversation history — in multi-turn chats, the entire thread gets re-sent with each new message. A 10-turn conversation means your 11th message is billed for all prior context.
Claude, GPT-4, and Gemini all tokenize differently. The same 500-word prompt can vary by 15% across platforms because of how they handle whitespace, punctuation, and non-English characters.
How to audit what you’re actually being billed for
Most API dashboards show total tokens per request, but they don’t break out input vs. output vs. overhead. Here’s how to measure it yourself:
Use the platform’s tokenizer tool. OpenAI offers a public tokenizer at platform.openai.com/tokenizer. Anthropic has a similar tool in their API docs. Paste your exact prompt—including system instructions and any JSON formatting—and compare the count to what your API response reports.
Log the token counts returned in API responses. Most platforms return usage.prompt_tokens and usage.completion_tokens in the response object. Store these alongside your request text in a database or CSV. After a week, calculate the average overhead: total billed tokens minus your estimated input/output length.
Test with and without function schemas. If you’re using structured outputs or tool calling, run the same prompt twice—once with the schema, once without. The delta is what the schema costs you per request. If it’s 200+ tokens and you’re making 10,000 requests a month, that’s $4–$12 in silent overhead (depending on the model).
The non-obvious fix: prompt caching
Some platforms now offer prompt caching, which stores repeated sections of your input and bills them at a fraction of the normal rate. Anthropic introduced this in mid-2026; OpenAI has a similar feature for GPT-4 Turbo.
If you’re sending the same system instructions, reference documents, or function schemas with every request, caching can cut your effective input cost by 50–90%. But it only works if the repeated section is identical across requests—down to the character. A single whitespace difference breaks the cache.
Here’s the trick: template your prompts so the static parts come first, and the variable parts (user input, dynamic data) come last. Cache the prefix. For example:
- Cached prefix: system instructions + JSON schema + reference examples (1,500 tokens)
- Variable suffix: user-provided text (200–500 tokens)
On a cached request, you’re only billed full-rate for the 200–500 variable tokens. The 1,500-token prefix costs ~10% of normal. Over 10,000 requests, that’s the difference between $150 and $18.
When token bloat doesn’t matter—and when it does
If you’re running a few dozen requests a day for summarization or drafting, the overhead is noise. You’ll spend more time auditing than you’ll save.
But if you’re processing user-generated content at scale—classifying support tickets, generating product descriptions, scoring leads—token bloat compounds fast. A 20% miscalculation on 50,000 requests per month is $200–$400, depending on the model.
The fix isn’t switching platforms. It’s measuring what you’re actually being charged for, trimming the fat (verbose system instructions, redundant examples), and caching anything that repeats.
Want more breakdowns like this? Subscribe to One Two Three Send—we explain one under-documented tool behavior every week, with the numbers platforms don’t advertise.
Heads up — some links in this article are affiliate links. If you sign up through them, we may earn a small commission at no extra cost to you. We only recommend tools we use ourselves.