Prompt Caching in 2026: Cut Your LLM API Costs by Up to 90%

A typical RAG chatbot sends the same 8,000-token system prompt and document set with every single user message. At Claude Sonnet 4.6's standard input rate of $3 per million tokens, that is $24 per million messages just in repeated context. With prompt caching enabled, those repeated tokens cost $0.30 per million. Same output, same latency or better, 90% cheaper.

Prompt caching is not a niche optimization. It is the default state you should be in for any production LLM workflow where a prompt prefix repeats across requests. Every major provider supports it in 2026 - Anthropic, OpenAI, and Google Gemini - and the implementation ranges from zero code changes (OpenAI) to a single extra field in your request body (Anthropic).

What Is Prompt Caching?

When an LLM processes your prompt, it computes attention key-value (KV) tensors for every token. Prompt caching stores those computed tensors server-side so that if your next request starts with the same prefix, the model skips recomputing them and loads the cached result directly. The output is byte-identical; only the prefill step was shortcut. Latency typically drops 30-80% for cache hits because prefill is often the slowest part of a request.

The Ordering Rule - Get This Wrong and Nothing Caches

Before touching any code, understand this rule: static content must come before dynamic content in your prompt.

Caching works by matching a prefix. The moment the prefix diverges from the cached version, nothing after that point can be cached. The correct order is:

Bash
Copy
1. Tool definitions (static)
2. System prompt (static)
3. Retrieved documents / knowledge base (semi-static)
4. Few-shot examples (static)
5. Conversation history (dynamic)
6. Current user message (dynamic)

The most common mistake is a system prompt that includes a timestamp, request ID, or user-specific greeting. A single variable string at position 1 means 0% cache hits. Move all dynamic content to the end, after everything static.

Anthropic Claude - cache_control Field

Claude's prompt caching is opt-in. You mark which content should be cached by adding "cache_control": {"type": "ephemeral"} to specific content blocks.

Pricing (June 2026)

Cache writes cost 1.25x the base rate. The break-even is after two reads - every read beyond that costs 90% less. Default TTL is 5 minutes, workspace-scoped. Prompt caching and Anthropic's Batch API stack: Batch gives 50% off all tokens, combine it with cache reads and you reach 95% savings on the repeated portion.

Python
Copy
import anthropic

client = anthropic.Anthropic()

KNOWLEDGE_BASE = """[Your 5,000-token product documentation here]"""

def ask_question(user_question: str, conversation_history: list) -> str:
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        system=[
            {
                "type": "text",
                "text": "You are a helpful support assistant.",
            },
            {
                "type": "text",
                "text": KNOWLEDGE_BASE,
                "cache_control": {"type": "ephemeral"},
            },
        ],
        messages=[
            *conversation_history,
            {"role": "user", "content": user_question},
        ],
    )

    usage = response.usage
    print(f"Cache read tokens: {usage.cache_read_input_tokens}")
    print(f"Cache write tokens: {usage.cache_creation_input_tokens}")

    return response.content[0].text

On the first call, cache_creation_input_tokens is non-zero and cache_read_input_tokens is 0. On every subsequent call within the TTL window the numbers flip and you pay 10% of normal input cost.

OpenAI - Automatic, Zero Code Changes

OpenAI's prompt caching is automatic on all current flagship models. No marker, no flag. OpenAI checks whether your request prefix matches a recent cached prefix server-side and applies the discount automatically.

Pricing (June 2026)

Every model in the GPT-5.x family delivers 90% off on cache reads. The gpt-5.5-pro and gpt-5.4-pro variants are pro-tier research models without public caching. For production use, gpt-5.4 at $2.50/M base is the recommended default - same 90% cache discount as GPT-5.5 at half the price. gpt-5.4-nano at $0.20/M makes high-volume apps with long repeated prompts extremely cost-efficient.

Python
Copy
from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[
        {"role": "system", "content": LONG_SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
)

usage = response.usage
cached = usage.prompt_tokens_details.cached_tokens
total_input = usage.prompt_tokens

print(f"Cache hit rate: {cached / total_input:.1%}")

OpenAI only caches prefixes of at least 1,024 tokens. Under that threshold you will never see a hit. Cache retention defaults to 24 hours for most accounts. If your cached_tokens field is consistently 0, check whether dynamic content has leaked into your system prompt prefix.

Google Gemini - Explicit Context Caching

Gemini lets you explicitly create a named cache object, pay for its storage by the hour, and reference it by ID in requests. More setup, more control over cache lifetime.

Pricing (June 2026)

The storage cost is the catch. A 100,000-token Flash cache costs $0.10/hour. It pays for itself quickly at high request volume, but for low-traffic workloads run the math first. Minimum cache size is 4,096 tokens. Gemini also supports implicit caching (like OpenAI's automatic mode) for paid projects, with cache reads at 10% of base rate and no code changes.

Python
Copy
import datetime
from google import genai
from google.genai import types

client = genai.Client()

# Create the cache once, reuse for hours
cache = client.caches.create(
    model="models/gemini-3-flash-preview",
    config=types.CreateCachedContentConfig(
        display_name="product_docs_cache",
        system_instruction="You are a code review assistant.",
        contents=[types.Content(role="user", parts=[types.Part(text=LARGE_DOCUMENT)])],
        ttl=datetime.timedelta(hours=1),
    ),
)

def review_code(code_snippet: str) -> str:
    response = client.models.generate_content(
        model="models/gemini-3-flash-preview",
        contents=code_snippet,
        config=types.GenerateContentConfig(cached_content=cache.name),
    )
    return response.text

# Delete when done to stop storage billing
client.caches.delete(name=cache.name)

Provider Comparison

Quick decision guide:

• Already on OpenAI: fix your system prompt ordering today and migrate to gpt-5.4 for the 90% cached discount.
• On Anthropic with long system prompts or RAG docs: add cache_control to the static blocks and watch cache_read_input_tokens.
• On Gemini for large-document analysis: use explicit context caching and delete caches after use.

Open-Source Tools to Reduce LLM Costs Further

Prompt caching handles repeated prefixes. These open-source tools attack costs from different angles - compressing what you send, caching semantically similar responses, and routing requests to cheaper models automatically.

Recommended stack by workload:

• Simple chatbot: prompt caching + LiteLLM (routing + budget cap) + Langfuse (visibility)
• High-traffic service: prompt caching + Bifrost (gateway with semantic cache)
• RAG pipeline: OpenProvence (prune retrieved docs) + prompt caching + Headroom
• Mixed-difficulty queries: RouteLLM (route easy to cheap) + prompt caching on the cheap model

Best Use Cases

• RAG applications - Multiple users querying the same knowledge base means the same retrieved documents get cached across requests. Each cache hit saves 90% on that document block.
• Long-context agents - A ReAct-style agent with 20 tool definitions and a planning prompt burns significant input tokens per step. Cache the static portion and every agent step gets cheaper.
• Multi-turn chatbots - Cache the system prompt and fixed product context. Only new conversation turns bill at full rate.
• Code review pipelines - Cache the codebase once and run multiple analysis passes (security, style, docs) against it. This is exactly what Gemini's explicit caching was built for.

Common Gotchas

• Temperature does not affect cache hits. The cache stores KV tensors for the prompt, not the output.
• Different models do not share caches. A claude-sonnet-4-6 cache cannot be hit by claude-haiku-4-5.
• Model weight updates invalidate caches. Rare for stable model IDs, but it happens.
• Streaming works fine. Cache write tokens appear in the usage object on the final stream event.
• Gemini storage costs accumulate. For occasional batch jobs, delete the cache immediately after use.
• OpenAI's 1,024-token threshold is real. A 900-token system prompt never caches, regardless of consistency.

Conclusion

Prompt caching is the fastest cost optimization available to any team running LLMs in production. It requires no model changes, no quality tradeoffs, and minimal implementation overhead.

The one thing that requires thought is prompt structure. Static content first, dynamic content last. Get it wrong and the cache never hits. Get it right and 60-90% of your input tokens cost a fraction of what they did before.

Start with the provider you are already using. Add cache_control to your Anthropic system prompt today and check cache_read_input_tokens tomorrow morning. Or track OpenAI's cached_tokens field this week. The numbers will make the case for going deeper.