Token Re-Hydration & Repeated System Prompts
The Vulnerability: A massive 1,800-token system prompt packed with static JSON schemas, extraction guidelines, and product documentation was prepended synchronously on every single turn of conversation. For a 6-turn session, 10,800 identical tokens were re-billed to OpenAI without any cache participation.
The Engineering Fix: We restructured prompt layouts to take advantage of exact prefix caching, isolated transient runtime context to the final tail block, and deployed a Cloudflare KV cache for idempotent document extraction hashes.