Prompt Caching Explained: How It Works, Pricing and Hit Rate

Look closely at any serious LLM bill and you will notice something counterintuitive: the expensive part is usually not what the model writes. It is the input you resend on every single request — the system prompt, the tool definitions, the reference documents, the ever-growing conversation history. Prompt caching exists to fix exactly that. Repeated prefixes are cached, and when a later request hits the cache, those tokens are billed at a fraction of the normal input price while time-to-first-token drops as well.

This guide covers how prompt caching works, how Anthropic and OpenAI approach it differently, how to think about the pricing, how to structure prompts so the cache actually hits, and why it matters so much for tools like Claude Code. If you want a refresher on token math first, see the token cost calculator guide.

What Prompt Caching Actually Caches

Before a model generates anything, it processes the entire input and computes an internal state for every token. If two requests start with exactly the same content, that work can be reused. That is the whole idea: the cache stores the processed input prefix, not the model's answer.

Three properties follow from that:

• Prefix-only matching. The cache matches from the very first token. Change one character in the middle and everything after it is a miss.
• No effect on output quality. With or without a hit, the model sees identical input. Answers are not stale or degraded.
• Limited lifetime. Cached prefixes expire after a period of inactivity and must be written again.

Every optimization later in this article is a consequence of the prefix rule.

Anthropic: Explicit Caching with cache_control

Claude models use explicit caching. You mark where the cacheable prefix ends with a cache_control block, and everything before that point — in the order tools, then system, then messages — becomes a cache entry.

curl https://zivv.pro/v1/messages \
  -H "x-api-key: sk-your-zivv-key" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "max_tokens": 1024,
    "system": [
      {
        "type": "text",
        "text": "You are a senior code reviewer... (several thousand tokens of guidelines)",
        "cache_control": {"type": "ephemeral"}
      }
    ],
    "messages": [
      {"role": "user", "content": "Review this function: ..."}
    ]
  }'

Things worth knowing:

• A small number of breakpoints. The official docs currently allow up to four per request, which lets you cache tools, system prompt and conversation history as separate layers.
• A minimum length. Very short prefixes are not cached. The threshold varies by model (roughly one to a few thousand tokens); check the official documentation for the model you use.
• A short default lifetime. The default TTL is about five minutes and is refreshed on every hit. A longer TTL option exists at a higher write cost.
• Observable usage fields. The response reports cache_creation_input_tokens (written this time) and cache_read_input_tokens (served from cache). These fields are the only reliable way to know whether you hit.

Through Zivv, the Anthropic-native endpoint is https://zivv.pro and the request body is identical to the official API, so cache_control works as-is. If you need precise cache control, call /v1/messages natively rather than reaching Claude through the OpenAI-compatible format, where cache markers do not translate cleanly.

OpenAI: Automatic Caching

GPT models take the opposite approach: caching is automatic and requires no code changes. Once a prompt exceeds a minimum length (1,024 tokens per OpenAI's docs), the platform tries to reuse matching prefixes from recent requests.

You still have two levers:

• Observe hits via usage.prompt_tokens_details.cached_tokens in Chat Completions, or input_tokens_details.cached_tokens in the Responses API.
• Improve routing with a cache key parameter such as prompt_cache_key, which makes requests that share a prefix more likely to land on the same cache. Useful when many users share one system prompt.

from openai import OpenAI

client = OpenAI(api_key="sk-your-zivv-key", base_url="https://zivv.pro/v1")

resp = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {"role": "system", "content": LONG_STATIC_SYSTEM_PROMPT},
        {"role": "user", "content": "Summarize this weekly report"},
    ],
)
details = resp.usage.prompt_tokens_details
print("cached:", details.cached_tokens if details else 0)

Side-by-Side Comparison

Gemini models have a comparable mechanism (newer models support implicit caching). The playbook is the same: keep the prefix stable and watch the usage fields.

How to Think About Pricing

The publicly documented ballpark: on Claude, a cache write costs about 1.25x the base input price at the default TTL, and a cache read about 0.1x. On OpenAI, cached input is discounted anywhere from roughly half price to a tenth of the price depending on the model. Treat these as orientation only — the official price lists and the [Model Hub](/model-hub) are authoritative.

Zivv bills as group multiplier times the official model price, per actual tokens. Cache reads, cache writes and normal input are each priced from their own official rate, so the savings you get from caching upstream carry through to your Zivv bill.

A quick estimate. Say your system prompt plus project docs come to 20,000 tokens, each user turn adds 500, and you run 20 turns inside the cache lifetime:

• Without caching, all 20 turns pay full input price on those 20,000 tokens — 400,000 tokens of repeated input.
• With caching, the first turn writes the prefix at about 1.25x, and the next 19 turns read it at about 0.1x. The repeated portion ends up costing a little over a tenth of the uncached version.

The rule of thumb: the longer the prefix, the more often it is reused, and the shorter the gap between requests, the more caching pays off. A long prefix used exactly once costs slightly more with caching because of the write premium — which is precisely why Anthropic makes caching opt-in.

Seven Ways to Raise Your Hit Rate

1. Static first, dynamic last. Tools, system prompt and reference docs go at the top; the user's question and anything time-dependent go at the end.
2. Keep timestamps out of the system prompt. A line like "current time: 14:32" changes every request and invalidates everything after it. If you need it, put it in the latest user message.
3. Serialize tools deterministically. If tool definitions are generated at runtime, sort them and keep JSON key order stable.
4. Append to history, never rewrite it. Editing or re-summarizing earlier messages breaks the prefix.
5. Use multiple breakpoints. Separate rarely-changing tools, occasionally-changing project notes and the growing conversation so a local change does not wipe the whole cache.
6. Batch requests together. With a lifetime measured in minutes, a batch job with long pauses between items will keep missing.
7. Avoid switching models mid-session. Caches are per model; a switch starts from zero.

Then verify: send the same request twice and confirm that cache_read_input_tokens or cached_tokens on the second response is well above zero.

Prompt Caching in Claude Code

Claude Code is one of the biggest beneficiaries of prompt caching. Every request carries a large, mostly stable prefix: the system prompt, all tool definitions, your CLAUDE.md, files it has read, and the growing conversation. Claude Code places cache breakpoints automatically — there is nothing to configure.

What that means in practice:

• Long sessions are cheaper than they look, because most of the prefix is billed at the cache-read rate.
• Coming back after a long break costs a bit more on the first turn, since the cache has expired and must be rewritten.
• Switching models or adding and removing MCP tools mid-session forces a rebuild. Do it when you need to, not casually.
• `/compact` changes the prefix. The first turn after compaction rewrites the cache, but every turn after that is cheaper.

With Zivv, Claude Code only needs ANTHROPIC_BASE_URL=https://zivv.pro and ANTHROPIC_AUTH_TOKEN. It speaks the native Anthropic protocol, so caching behaves exactly as it does upstream. See the Claude Code Windows setup guide or the Claude Code docs, and heavy users should look at the Claude MAX group. For broader savings, read AI cost optimization.

FAQ

Q: Does prompt caching make answers stale or worse? A: No. Only the processed input prefix is cached. The model generates a fresh answer from the full input every time, and output quality is identical with or without a hit.

Q: How do I confirm the cache is hitting? A: Read the usage fields: cache_read_input_tokens for Claude, cached_tokens for OpenAI. The Zivv console's usage details also reflect how input tokens break down.

Q: Can I use caching when calling Claude through the OpenAI-compatible endpoint? A: Cache markers do not map cleanly across protocols, so behavior depends on the gateway's translation. For precise control, use the native Anthropic endpoint at https://zivv.pro/v1/messages.

Q: Can other users read my cached prompts? A: Caches are isolated per organization at the provider, and a hit requires an exact prefix match. Zivv itself does not store prompt or response bodies; it records only token usage for billing.

Q: Cache writes cost more — can caching lose money? A: Slightly, if a prefix is used only once. If the same prefix is reused two or more times within its lifetime, caching comes out ahead.

Q: Do I need to turn caching on in Claude Code? A: No, Claude Code uses prompt caching by default. Your job is to keep sessions continuous and avoid unnecessary model switches.

Wrapping Up

Prompt caching is the cheapest cost optimization you can make: restructure your prompt slightly — or change nothing at all — and long-context workloads get dramatically cheaper on input. Put static content first, watch the usage fields, and keep sessions continuous. Sign up for Zivv and verify cache hits on both Claude and GPT with a single key.