Learn AI Create the Future — with Pranay
Main site

Caching and Cost Control

Caching model predictions

If the model has already answered this exact question, hand back the saved answer instead of running the model again.

Read these first

On this page 8
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. A real example you have seen
  6. The honest part
  7. Remember this
  8. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

A prediction cache saves the model's past answers, so the exact same question never has to be answered twice.

The analogy you have already lived

A school office keeps a folder of answers to questions every parent asks. "What time does school start?" The clerk does not walk to the principal's office each time. They read the answer off a printed sheet.

The principal only gets asked something new. Anything repeated gets handled at the front desk, in seconds, by someone who did no new thinking at all.

A prediction cache is that folder, sitting in front of your model.

Why it exists

Running a model costs time and, often, real money — a GPU-second, or a paid API call. Many real systems get asked the same question, or something very close to it, again and again.

A weather app shows the same forecast to a thousand people who opened it in the same minute. A translation tool sees "thank you" a thousand times a day. Paying the full model cost for every one of those wastes money. That work was already done once.

How it works

   request comes in: "translate 'thank you' to Hindi"
              |
              v
   have we seen this EXACT question before?
        |                          |
       yes                         no
        |                          |
   hand back the                run the model
   saved answer                 SAVE the answer
   (fast, free)                 (slow, costs money)
              \                  /
               v                v
            answer goes back to the caller

The saved answers live in a cache — a plain-English word for "a small, fast storage area for things you might need again soon."

A real example you have seen

Google Translate gives you an instant answer for a phrase millions of people have already asked it to translate. Weather apps do not re-run a forecast model for every single person who opens the app in the same hour. Autocomplete on your phone keyboard has seen "how are" a billion times, and does not think from scratch each time.

The honest part

This only helps when the same question genuinely repeats. A system where every input is unique gets nothing from this kind of cache. Every photo is a different photo. Every sensor reading is a fresh number. Knowing whether your traffic repeats is worth checking before you build one.

Remember this

  • A prediction cache stores the model's past answers for exact repeated questions.
  • It saves both time and, for paid models, money.
  • It only helps when questions genuinely repeat. Check that they do, first.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install cachetools

A cache in front of a slow function

cache_demo.py
import time
from cachetools import LRUCache

# Stands in for a real model: slow, and gives the same answer for the same input.
def slow_model(x: int) -> int:
    time.sleep(0.05)  # pretend this is 50ms of real GPU work
    return x * x

cache = LRUCache(maxsize=128)  # holds at most 128 answers, oldest evicted first

def predict(x: int) -> int:
    if x in cache:
        return cache[x]
    result = slow_model(x)
    cache[x] = result
    return result

for x in [4, 7, 4, 9, 7, 4]:
    start = time.perf_counter()
    y = predict(x)
    ms = (time.perf_counter() - start) * 1000
    print(f"predict({x}) = {y}   {ms:6.1f} ms")

print("cache now holds:", dict(cache))
Output
predict(4) = 16     50.3 ms
predict(7) = 49     50.0 ms
predict(4) = 16      0.0 ms
predict(9) = 81     50.1 ms
predict(7) = 49      0.0 ms
predict(4) = 16      0.0 ms
cache now holds: {4: 16, 7: 49, 9: 81}

The exact milliseconds are a real measurement from this machine, and will vary slightly on yours. The pattern will not: first sight is slow, repeats are near zero.

Line-by-line walkthrough

LRUCache(maxsize=128). LRU stands for least recently used. When the cache is full, it deletes the answer nobody has asked for in the longest time, making room for a new one. Without a limit, a cache can grow forever and eat all your memory.

The cache key is x itself. For a real model, build the key from every input that affects the answer — every field, in a fixed order. Two requests differing in even one ignored space, or one field order, can silently miss the cache. The model reruns needlessly. Build the key deliberately.

Nothing here is safe for changing data. Say slow_model could return a different answer for the same x later — a model gets updated, a price changes. This cache would happily serve a stale answer forever. See the mistakes below.

Common mistakes

No expiry on data that changes. A cache with no time limit assumes the true answer never changes. Add a TTL — time to live, how long an entry is trusted before it is recomputed — whenever the underlying answer can drift.

Caching by accident across users. A recommendation cached under a key that forgets to include the user ID hands one person's results to someone else. Always ask: does this key uniquely describe everything that should affect the answer?

No size limit. An unbounded cache is a memory leak with a delay. LRUCache(maxsize=...) or a real cache server like Redis both solve this. A plain Python dict used as a cache does not.

Caching errors. If slow_model raises an exception, do not cache the failure. The next legitimate request deserves a real attempt, not a stored crash.

Try it yourself

Add a stats counter that tracks hits and misses, and print the hit rate after the loop. Then run 1,000 random integers between 1 and 20 through predict and see how high the hit rate climbs.

What to learn next

Researcher — Mathematics and papers.

Expected latency with a cache

Let $h$ be the hit rate — the fraction of requests the cache can answer — measured between 0 and 1. Let $L_{\text{hit}}$ be the time to serve a cache hit and $L_{\text{miss}}$ the time to run the model. Expected latency is

$$\mathbb{E}[L] = h \cdot L_{\text{hit}} + (1 - h) \cdot L_{\text{miss}}$$

Because $L_{\text{hit}} \ll L_{\text{miss}}$ in almost every real system, expected latency is dominated by $(1-h)$, the miss rate. Doubling the hit rate from 0.5 to 0.9 does far more for average latency than any realistic speedup to the model itself.

The same identity holds for cost when $L$ is replaced with dollars per request — which is why caching is usually the single highest-leverage cost lever available before touching the model.

Eviction policies

When a cache is full, something has to be removed to make room. The policy choice matters more as memory gets scarce.

  • LRU (least recently used) — evict the entry unused for the longest time. Cheap, and a strong default when recent requests predict near-future ones.
  • LFU (least frequently used) — evict the entry with the fewest total hits. Better when a stable set of "hot" items dominates traffic, worse at adapting when popularity shifts.
  • Belady's algorithm — evict whichever entry is needed furthest in the future. Provably optimal (Belady, 1966), and impossible to run in production because it requires knowing the future request stream. It is the yardstick every practical policy is measured against.
  • ARC (Adaptive Replacement Cache) — Megiddo and Modha (2003) combine recency and frequency and tune the balance between them online, closing much of the gap to Belady's algorithm without foreknowledge.

Where caching interacts with correctness

A prediction cache is a statement that the function being cached is pure — same input, same output, forever — which is rarely fully true for a model that gets retrained. Every cached system needs an explicit invalidation strategy (see cache invalidation for RAG for the harder version of this problem), or a bounded TTL that accepts some staleness as the cost of the savings.

Papers

  • Belady, A Study of Replacement Algorithms for a Virtual-Storage Computer, IBM Systems Journal, 1966
  • Megiddo and Modha, ARC: A Self-Tuning, Low Overhead Replacement Cache, FAST 2003

What to learn next