Prompt Caching: Cut LLM Costs Without Compromise

Every startup building AI features discovers the same uncomfortable truth after their first production month: LLM API costs scale with users in ways that weren’t obvious from the demo.

You’re paying for tokens on every call. And most of those tokens are the same tokens — your system prompt, your retrieved documents, your product instructions — repeated verbatim across thousands of requests. You’re paying to process the same text over and over, from scratch, every time.

Prompt caching is the fix. It’s not widely discussed, but every major AI provider now supports it. Done right, it cuts input token costs by 60–90% on affected calls without changing model behavior at all.

The Problem Prompt Caching Solves

Why production AI costs spike

In development, you make a few hundred calls. The cost is negligible. In production, a moderately active user base might generate 50,000 calls per day. Suddenly the math looks different.

The expensive part isn’t usually the output — it’s the input. And production prompts tend to be long. A RAG-powered assistant might send 2,000 tokens of retrieved documents plus 500 tokens of system instructions with every request. Only the user message (maybe 50–100 tokens) changes between calls. You’re paying for 2,500 tokens per call when the marginal new information is 75 tokens.

Prompt caching breaks that pattern. You pay full price the first time a chunk of context is processed. Subsequent calls that reuse the same context pay a cache-read rate — typically a fraction of the normal price.

What gets cached

Not all prompt content is equally cacheable. Static content that doesn’t change between users or sessions is the high-value target:

• System instructions and product persona definitions
• Fixed document context (terms of service, product documentation, knowledge base articles)
• Tool and function definitions in function-calling setups
• Long few-shot examples that remain constant across users

Dynamic content — the user’s message, session-specific history, real-time data — isn’t cacheable by nature, but it’s also usually the smallest part of your prompt.

How Each Provider Implements It

Anthropic (Claude)

Anthropic’s approach is explicit. You mark boundaries of what you want cached using cache_control markers in the message structure:

messages=[
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": LARGE_SYSTEM_CONTEXT,
                "cache_control": {"type": "ephemeral"}
            },
            {
                "type": "text",
                "text": user_message
            }
        ]
    }
]

The cache lives for 5 minutes by default and resets each time it’s hit. Cache reads cost 10% of the normal input token price — a 90% discount. Cache writes (the first call that populates the cache) cost 25% more than normal. So the break-even is the second call within that window.

OpenAI

OpenAI’s caching is automatic for prompts exceeding 1,024 tokens. No code changes required — the API handles it transparently. Cached prefix tokens are billed at 50% of the standard input rate.

The catch: you don’t control what gets cached or see whether a cache hit occurred (without inspecting response metadata). Less predictable than Anthropic’s explicit model, but it’s the lowest-friction starting point.

Google (Gemini)

Google takes the most explicit approach. Gemini’s Context Caching API requires you to create a cache object separately, then reference it by ID in subsequent requests:

cache = genai.caching.CachedContent.create(
    model="gemini-1.5-pro-001",
    contents=[large_document_content],
    ttl=datetime.timedelta(hours=1)
)
response = model.generate_content(contents=user_query, cached_content=cache)

You set a TTL, pay for cache storage while it’s alive, and pay reduced rates for cache reads. More management overhead, but also more control — useful when caching large documents across many concurrent users who all share the same base context.

When Prompt Caching Actually Makes a Difference

The savings compound when your prompt architecture has these characteristics:

High static-to-dynamic ratio. If 70% of your average prompt is fixed (system prompt plus document context) and 30% is dynamic (user query plus recent history), caching the fixed portion cuts 70% of your input tokens on every cache-hit request.

High request volume. The first call per cache window is always full price. The break-even on cache write overhead is the second identical call. At low volume, the savings are marginal. At production scale, the math shifts heavily in your favor.

Multi-tenant applications with shared context. If many users share the same base context — a product knowledge base, a set of policy documents, API documentation — that context can be cached once and reused across all of them.

If your average prompt is mostly dynamic — conversational responses with little fixed context, or prompts that change completely per user — caching won’t help much. It’s a tool for specific architectures, not a universal cost fix.

What to Watch Out For

Cache invalidation follows strict rules. For Anthropic, the cache key is the exact byte sequence of the cached content. A single character change — even a trailing space — creates a new cache entry and resets the clock. Build your prompts so static sections genuinely stay static. Dynamic content belongs at the end of the context, after any cached blocks.

Cold-start latency on the first request matters in latency-sensitive features. Cache writes carry slight overhead. If you’re building a real-time chatbot where every millisecond counts, measure first-call latency and decide whether the tradeoff is acceptable.

Don’t try to cache short content. The mechanics require a minimum context length (1,024 tokens for OpenAI, 2,048 for Anthropic). If your system prompt is 300 tokens, caching won’t activate — you need to think about what else belongs in that shared context block.

Making the Economics Work

Prompt caching is one layer in a broader AI API cost control strategy. Combined with model routing (cheaper models for simpler tasks), output caching (storing identical responses at the application layer), and prompt compression, a well-architected AI product can run at a fraction of naive per-call pricing.

The teams that get this right build it into their architecture from the start, not as a retrofit. When we build AI-powered products at quickdev, prompt structure and caching strategy are first-class decisions — because the cost model of a production AI feature looks very different from a prototype, and the right time to fix it isn’t six months after launch.

---

Yaniv Amrami is founder of quickdev. He has helped Israeli startups build and optimize AI products since the early days of the LLM API ecosystem.