Prompt Caching in Coding Agents: The Silent Miss That Bills Full Price Again

You turn on prompt caching for your coding agent, the cost drops, everyone screenshots the dashboard. Weeks later someone reorders the tools list in a "cosmetic" refactor, or drops a timestamp into the system block to make logs easier to read, or starts injecting project memory at the top of the prompt. None of these changes fails a test. None of them returns an error. The API keeps answering 200. What changes is that every call goes back to paying for the entire prefix as if the cache never existed — and that only shows up on the invoice, weeks later, when someone asks why cost per task crept back up.

That's the problem this article is about — not blowing through a spend limit (we covered that in cost cap and circuit breaker), and not proving which prompt version ran on a given task (that's the versioned prompt). It's a harmless formatting change silently invalidating the token cache, with no error log and no budget alert, because the "failure" is indistinguishable from a normal call — the only difference lives in numeric fields inside a usage object that almost nobody checks in production.

How the cache decides yes or no (and why it's all-or-nothing)

Prompt caching works by exact prefix matching. The provider stores the result of processing a set of tokens; on the next call, it compares the start of the payload byte-for-byte against what's cached. If it matches up to a certain point (the "breakpoint"), it reuses that work; from the first divergent byte onward, everything gets reprocessed from scratch — including anything after it that would otherwise have been identical. There's no partial "close enough" cache.

The order in which the provider assembles that prefix matters more than anything else:

• Anthropic builds the prefix as tools → system → messages. Any change to a tool's name, description, parameters, or order invalidates everything downstream, including system. Caching here is opt-in via cache_control: you can set an automatic top-level breakpoint or mark specific blocks manually, up to 4 breakpoints per request.
• OpenAI renders a prefix that includes hidden system + tools + developer + message history. Changing tools (name, schema, description, order), the model, parallel_tool_calls, text.format, or text.verbosity breaks the cache. Caching here is automatic by default (implicit); explicit breakpoints only exist from the GPT-5.6+ family onward, and without an explicit marker the provider decides on its own where to cut — which doesn't always line up with the end of your static content.

That philosophical split — Anthropic opt-in and granular, OpenAI automatic and "trust us" — explains why the two providers fail in different ways. On Anthropic, you forget to move cache_control to the right block. On OpenAI, the provider breakpoints the end of the conversation, which shifts every turn, and never reuses the static prefix sitting further back.

The three causes that actually show up

Reordering tools. Teams do this without thinking twice: sorting alphabetically, moving the newest tool to the top, generating the list from a Python dict whose iteration order isn't guaranteed across runs. Any of these rewrites the entire prefix. Fix: serialize tools with a fixed canonical order (by name, always) before building the payload — never rely on dict insertion order or an implicit database ORDER BY. Production check: compare the hash of the tools array you sent against the hash of the cached array; if they diverge without an intentional tool change, that's a serialization bug, not product drift.

A timestamp (or anything that changes every call) inside the breakpoint block. It's common to stuff "current time" or a "session id" into system to give the model context. If that block sits right before the cache cut, every call is a write and none is a read. Fix: move dynamic content (timestamp, project memory, session state) to after the cut point — the cacheable prefix should hold only what's identical across calls: tools and stable instructions. Production check: a fixture test that runs the same prompt twice with only the timestamp varying; if the second call shows no cache read, the breakpoint is in the wrong place.

Project memory injected into the static prefix. This is a different problem from "who assembles the context," which we already covered in project memory for coding agents — here the issue isn't where the memory comes from, it's where it sits in the payload. If memory lands before the breakpoint (or reorders lines within system), every new task produces a slightly different prefix and the cache never converges. Fix: treat project memory as post-breakpoint content, with stable formatting (same field order, same json.dumps with sort_keys=True) whenever it does need to go before the cut. Production check: log the token size of the pre-breakpoint block on every call; size variation there is a sign of dynamic content leaking into the static prefix.

Measuring hit rate from the usage fields

The API response always tells you what happened — just not in the form you're used to checking (status code). You have to read the usage fields.

On Anthropic, the usage object carries cache_read_input_tokens, cache_creation_input_tokens, and input_tokens (the last one counts only the tokens after the last breakpoint). Creation splits into ephemeral_5m_input_tokens and ephemeral_1h_input_tokens, which sum to cache_creation_input_tokens. If both cache fields come back zero, the call used no cache at all — neither read nor write — usually because the prefix fell below the minimum cacheable size.

On OpenAI, usage.input_tokens_details.cached_tokens shows how much was read from cache; from GPT-5.6+ onward there's also cache_write_tokens (don't invent that field on responses from earlier models — caching there is read-only and implicit, with no write charge).

A silent miss is this, in one line: HTTP 200 with cache_read (or cached_tokens) equal to zero, while write/uncached is greater than zero. Nothing broke. The agent answered correctly. Only the price went back to full.

Minimal fixture of a silent miss (Anthropic shape, one turn after the first):

{
  "cache_read_input_tokens": 0,
  "cache_creation_input_tokens": 41230,
  "input_tokens": 812
}

If this were a healthy cache read, cache_read_input_tokens would be near 41230 and cache_creation_input_tokens would be zero. Here the entire prefix got rewritten — someone changed something before the breakpoint.

Computing the hit rate is trivial arithmetic, but it's worth keeping the function ready to reuse in CI and in alerting:

def hit_rate(read: int, write: int, uncached: int) -> float:
    den = read + write + uncached
    return 0.0 if den == 0 else read / den

CI that locks the static prefix before it reaches production

Measuring in production tells you after the fact. What actually prevents the problem is a CI test that hashes exactly what should be stable — tools and the fixed system, no timestamp, no memory — and fails if that hash changes without an explicit decision.

import hashlib
import json
import pathlib
import sys

ROOT = pathlib.Path(__file__).resolve().parent / "agent_prompts"

def canonical_prefix(tools, system: str) -> bytes:
    blob = json.dumps(
        {"tools": tools, "system": system},
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
    )
    return blob.encode()

def main() -> int:
    tools = json.loads((ROOT / "tools.json").read_text())
    system = (ROOT / "system.txt").read_text()
    if "T=" in system or "{{now}}" in system:
        sys.stderr.write("DYNAMIC MARKER IN STATIC PREFIX\n")
        return 2
    digest = hashlib.sha256(canonical_prefix(tools, system)).hexdigest()
    golden = (ROOT / "prefix.sha256").read_text().strip()
    if digest != golden:
        sys.stderr.write(f"STATIC PREFIX DRIFT {golden[:12]} -> {digest[:12]}\n")
        return 1
    print(f"PREFIX OK {digest}")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Stdlib only (hashlib, json, pathlib) — no Docker, no real LLM key, no AWS. This runs on any CI runner.

Against a stable tools.json and system.txt, the hash matches the recorded golden (03fd72daf7ab0ff42250b0f9a974bbb14d2533c93a8dc56fcd24aa509a02e8a1) and the script prints PREFIX OK. Reordering the two tools inside tools.json — without changing a single character of content, only the order of the entries — is already enough to produce:

STATIC PREFIX DRIFT 03fd72daf7ab -> 3635aaa3d313

with exit code 1. Undo the reorder and the script prints PREFIX OK again. It's the same kind of change flagged above as the most common cause of a silent miss — except now it's caught before the merge, not on next month's bill. The golden hash should only be updated in a PR that intends to break the cache (a tool swap, an instruction change); any other diff to that file is a warning sign, not routine maintenance.

Alerting on a hit-rate drop, without confusing it with TTL expiry

The CI hash catches configuration drift. It doesn't catch the case where the prefix is correct but the cache simply expired — that's expected, not a bug. So the production alert needs a threshold, and the threshold is team policy, not a vendor SLA.

THRESHOLD = 0.80  # local threshold, not a guarantee from any provider

def check(rows):
    alerts = []
    by = {}
    for r in rows:
        by.setdefault(r["conv_id"], []).append(r)
    for cid, rs in by.items():
        rs = rs[1:] if len(rs) > 1 else rs  # drop the first turn's write
        read = sum(x["read"] for x in rs)
        total = sum(x["read"] + x["write"] + x["uncached"] for x in rs)
        rate = 0.0 if total == 0 else read / total
        if total and rate < THRESHOLD:
            alerts.append({"conv_id": cid, "hit_rate": round(rate, 4), "total_input": total})
    return alerts

The first turn of any conversation is always a write — there's no cache before there's a first call. Dropping it from the calculation keeps the alert from firing falsely at the start of every session.

Before treating a one-off dip as an incident, remember that a single miss can just be TTL expiry: on Anthropic the cache lasts 5 minutes from the start of the request (refreshed on every hit); on OpenAI, from GPT-5.6+ onward, the documented minimum is 30 minutes. A miss after a quiet stretch from the agent is expected. A hit rate that drops and doesn't recover on subsequent calls is the signal that matters — that's when it's worth digging into tool reordering or dynamic content leaking into the static prefix.

What this costs in numbers, without vague generalizations

Official prices as of 2026-09-13, for a 100,000-token prefix, no output:

def prefix_cost_usd(tokens: int, uncached: float, hit: float, write: float | None = None, mode: str = "hit") -> float:
    price = {"uncached": uncached, "hit": hit, "write": write or uncached}[mode]
    return price * tokens / 1_000_000

# claude-sonnet-5, 100k tokens: hit $0.02, uncached $0.20

The number that usually surprises people isn't a single call, it's a full session. In the format OpenAI documents for GPT-5.6+ — ten calls against the same static prefix, the first a write and the following nine reads — the aggregate prefix cost comes out to 1.25 + 9 × 0.10 = 2.15× the price of one uncached call, versus 10× if all ten calls had been uncached. That's the gap between a cache genuinely working across a session and a cache that's "on" but rewriting every turn because the breakpoint sits in the wrong place — the second scenario costs exactly as if caching had been switched off, except nobody actually made that call.

Recommendation

If your coding agent already runs in production with more than one call per session, it's worth investing in all three layers: a prefix hash in CI (catches drift before the merge), reading the usage fields on every call (catches the silent miss in real time), and a hit-rate alert with a team-defined threshold (catches slow degradation that no single PR explains). None of the three replaces the others — the hash doesn't see TTL, a single usage field doesn't see a trend, and an alert without the hash won't tell you why the rate dropped.

Where DevDojo would pull back: if call volume per session is low (one or two per task) and the static prefix is small, the payoff from all this instrumentation may not be worth the effort — a simple log of cache_read_input_tokens per call, reviewed manually now and then, covers the risk just fine. The extra engineering pays off when the prefix is large (many tools, long instructions) and the session runs long enough for the cache to earn back the initial write several times over — which is exactly the profile of a coding agent that spends hours inside the same repository.

For the provider-specific details that don't fit here — minimum cacheable token counts, extended TTL options, the exact shape of cache_control — it's worth reading Anthropic's prompt caching docs, Anthropic's pricing page, OpenAI's prompt caching guide, OpenAI's pricing page, and xAI's pricing page before locking in your own threshold and minimum cacheable size.