Prompt caching
A vendor-side optimization that reuses computation for shared prompt prefixes across requests — billed at a 75–90% discount compared to fresh prompt tokens.
Prompt caching is the most important cost optimization of 2024–2026 for agent workloads. The pattern: a long system prompt (tool definitions, instructions, examples) stays the same across thousands of requests. The vendor caches the keys and values of those tokens and charges you a fraction of the input price to reuse them.
Anthropic shipped the canonical API in 2024; OpenAI and Google followed with similar (but differently priced) implementations. Cost savings vary: Anthropic discounts cached input tokens by 90%, OpenAI by 50%, Google by 75%. For an agent burning 10M tokens/day on shared system prompts, that is real money — often $1K+/month saved per agent.
Best practice in 2026: structure your prompts to put the stable, reusable content (system prompt, tool definitions, examples) at the top, and dynamic content (user query, recent state) at the bottom. The cache reuses tokens from the start until the first divergence.
Frequently asked
Does prompt caching change the output?+
No — caching is purely an inference-time optimization. The same prompt produces identical results whether cached or not. You just pay less and get faster time-to-first-token.
How long does the cache last?+
Vendor-dependent. Anthropic: 5-minute TTL by default, configurable to 1 hour at higher cost. OpenAI: minutes typically, no explicit guarantee. Google: minutes by default with explicit longer-cache APIs available.
Is prompt caching worth setting up?+
For any agent stack burning $200+/month in tokens, yes — even a 30-minute integration usually pays back in the first week. For one-off experiments, no.