Stop paying for the same answer twice — build a real response cache and see exactly how many redundant API calls it eliminates.
1The Economics of Repeated Questions
Internal chatbots see enormous overlap in questions — everyone eventually asks about PTO, benefits, and expense policies. Every one of those repeated questions running the full pipeline (embedding call plus generation call) is pure waste: the answer was already computed and is, by definition, identical for an identical question against unchanged documents.
2What to Cache — And What Not To
This lesson caches full responses keyed by exact question text, the simplest and safest caching layer. Production systems often add smarter caching too — semantic caching (matching near-identical questions, not just exact ones) and embedding caching (Module 2's approach). But exact-match response caching alone already eliminates a meaningful fraction of redundant cost in any real deployment, for almost no implementation complexity.
3Step-by-Step Breakdown
The Same Question, Asked Twice. In any real chatbot, users ask overlapping questions constantly — 'how much PTO do I get' gets asked by dozens of employees. Running the full embed-retrieve-generate pipeline fresh for an identical question you've already answered wastes real money and adds real latency for zero benefit.
Build a Response Cache. 4 questions are asked, but only 2 are actually distinct. Finish cached_pipeline(): if the question isn't already cached, run the (simulated) expensive pipeline, store the result, then return it — so a repeated question never touches expensive_pipeline() a second time.
Why check 'if question in cache' before calling the expensive pipeline, instead of always calling it and overwriting the cache?
- →Always calling the expensive pipeline defeats the entire purpose of caching — the check is what lets a repeated question skip the real API cost entirely instead of just re-storing the same result.
- →Python requires a membership check before writing to any dictionary.
Module 5 Complete — Ready to Ship. You've built, tested, and hardened a complete RAG chatbot: chunking, embeddings, retrieval, grounded generation, refusal handling, citations, injection defense, rate limiting, automated eval, and caching. Final lesson: what production deployment of this exact pipeline actually looks like.
Level Up 🚀
Advanced cheat sheets, SEO tricks, and interview prep for this topic.
Browser Support
Fully supported.
Fully supported.
Fully supported.
Fully supported.
Accessibility (A11y)
1A Cache Hit Should Never Change Response Timing Expectations Unpredictably
If cached responses return near-instantly while uncached ones take seconds, ensure loading indicators still behave predictably so users relying on consistent timing cues (including assistive tech users) aren't confused by wildly inconsistent response latency.
<div aria-busy="true">Thinking...</div>SEO Implications
- 1
Target 'LLM response caching' as a distinct, cost-focused search
Developers specifically search for caching once they've shipped a RAG system and started watching their actual API bill.
Best Practices
Cache Full Responses for Exact-Match Repeated Questions Before Anything Fancier
A simple exact-match cache is trivial to implement and immediately eliminates the most obviously wasteful redundant calls — build this first before investing in more complex semantic caching layers.
Frequent Bugs
Caching a response indefinitely even after the underlying source documents change, serving stale answers forever.
Invalidate or expire cache entries whenever the underlying document set is updated, or attach a time-based expiry so stale answers don't persist indefinitely.
Real-World Examples
Internal HR Chatbot at Scale
An HR chatbot serving 200 employees sees the same handful of policy questions asked repeatedly throughout the day — a simple exact-match cache cuts real API costs substantially without any change to answer quality.
if question in cache: return cache[question] # instant, free