tokens-usage
Guides
Count tokens before you send
Practical guides for preflight token counting, context window management, and cost estimation across OpenAI, Anthropic, and Google.
-
AI SDK streamText preflight
Agent loops grow fast. Count messages and tool history before streamText — not after onFinish.
Read guide → -
tiktoken vs OpenAI API
tiktoken.encode(prompt) ignores message framing. That gap causes context_length_exceeded errors in production.
Read guide → -
Claude dynamic max_tokens
Claude rejects requests when input + max_tokens > context limit. Measure first, then set max_tokens.
Read guide → -
Tool schema token overhead
Function schemas add 800–1,500 input tokens per turn. Measure tool overhead before your agent loop scales.
Read guide → -
AI SDK compaction triggers
Your AI SDK agent loop grows every step. This guide tells you when to compact — before safeMaxTokens hits zero and streamText returns a 400.
Read guide →