Learn AI Create the Future — with Pranay
Main site

Caching and Cost Control

Precomputing predictions

When you already know what inputs you will be asked about, score all of them ahead of time and serve the answers from a simple lookup.

Read these first

On this page 8
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. A real example you have seen
  6. The honest part
  7. Remember this
  8. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

If you already know what you will be asked, score every answer ahead of time. Serve requests from a lookup table, instead of running the model live.

The analogy you have already lived

A bakery does not wait for a customer to order bread before baking it. Every morning, before the shop opens, they bake a batch — knowing roughly what will sell.

When a customer arrives, the bread is already on the shelf. No one waits ten minutes for dough to rise. The slow work happened earlier, in bulk, while nobody was waiting.

Why it exists

Every earlier lesson in this section cached an answer after someone asked for it the first time. That still means the first person to ask waits for the slow, real computation.

Sometimes you do not need to wait for anyone to ask. Say you know your product catalogue, or your full list of customers. You can score every one of them overnight, before a single request arrives. Nobody is ever the "first person" who has to wait.

How it works

   BEFORE anyone asks (overnight, on a schedule):

   list of every known item
              |
   run the model on ALL of them, in one big batch
              |
   save every answer to a lookup table


   WHEN a real request arrives:

   "what's the score for item #4821?"
              |
   look it up in the table   (no model call at all)
              |
   answer back, instantly

This is called batch scoring. It works whenever the set of possible questions is known and small enough to score in advance. Not every product ever, but every product you currently sell, say.

A real example you have seen

A shopping app's "recommended for you" section appears the instant the page loads, for every one of its millions of users. Nobody's laptop is fast enough to compute personalised recommendations in the tiny moment between someone opening the app and the page appearing. Those recommendations were almost certainly computed for everyone, overnight, and are only being looked up now.

The honest part

This only works for questions you can list in advance. It cannot help with something genuinely new. A brand-new product added five minutes ago. A question nobody has asked the shape of before. Those still need a live model call. A real system usually needs both: a precomputed table for the known, common cases, and a live fallback for anything outside it.

Remember this

  • Precomputing runs the model before anyone asks, for everything you already know about.
  • Serving becomes a lookup, not a model call — nobody waits.
  • It only covers questions you can list in advance. New or unusual questions still need a live path.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn numpy

Scoring one at a time versus scoring the whole catalogue

precompute.py
import time
import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.RandomState(0)
X_train = rng.uniform(-3, 3, (500, 4))
y_train = (X_train.sum(axis=1) > 0).astype(int)
model = LogisticRegression().fit(X_train, y_train)

CATALOG_SIZE = 5000
catalog_ids = np.arange(CATALOG_SIZE)
catalog_features = rng.uniform(-3, 3, (CATALOG_SIZE, 4))

# ---- On-demand: score one item at a time, as requests happen to arrive ----
start = time.perf_counter()
for i in range(CATALOG_SIZE):
    model.predict_proba(catalog_features[i:i + 1])
on_demand_time = time.perf_counter() - start

# ---- Precomputed: score the whole catalogue once, store results for lookup ----
start = time.perf_counter()
all_scores = model.predict_proba(catalog_features)[:, 1]
lookup_table = dict(zip(catalog_ids.tolist(), all_scores.tolist()))
precompute_time = time.perf_counter() - start

# Serving a request afterwards is now a dictionary lookup.
start = time.perf_counter()
for i in range(CATALOG_SIZE):
    _ = lookup_table[i]
lookup_time = time.perf_counter() - start

print(f"on-demand, one row at a time: {on_demand_time*1000:8.1f} ms total")
print(f"precompute, whole catalogue:  {precompute_time*1000:8.1f} ms total")
print(f"lookup after precompute:      {lookup_time*1000:8.1f} ms total")
print(f"precompute is {on_demand_time/precompute_time:.1f}x faster than one-at-a-time scoring")
Output
on-demand, one row at a time:    223.0 ms total
precompute, whole catalogue:       0.8 ms total
lookup after precompute:           0.2 ms total
precompute is 265.5x faster than one-at-a-time scoring

That 265x is a real measurement from this machine, for this model and this catalogue size. Do not treat it as a universal constant. Most of the gap here comes from batching. scikit-learn scores 5,000 rows in one matrix operation far more efficiently than 5,000 separate one-row calls. On top of that sits the separate saving from never repeating work at serve time.

Line-by-line walkthrough

model.predict_proba(catalog_features), called once on the whole array. This is the whole trick. One call over 5,000 rows lets NumPy and scikit-learn use vectorised math, instead of paying Python's per-call overhead 5,000 separate times.

dict(zip(catalog_ids.tolist(), all_scores.tolist())). A plain Python dictionary is enough for a demo. A real system usually persists this to a fast key-value store or a database table. It then survives a restart, and the serving process can read it directly.

The final loop timing lookup_table[i]. This stands in for what a live request actually does after precomputing: a lookup, not a model call. It is measured separately from precomputing on purpose. They happen at different times — one overnight, one per user request.

Common mistakes

Precomputing for a catalogue that changes faster than the schedule. Say new items arrive every hour but scoring only runs nightly. New items then have no precomputed answer for up to a day. Pair this with a live fallback for anything missing from the table.

Storing the precomputed table somewhere the serving process cannot read cheaply. A lookup table that itself requires a slow network call defeats the purpose. It should live somewhere as fast as the win it is meant to deliver. Often that means an in-memory structure, or a local, fast key-value store.

Recomputing everything when only a little changed. If 50 out of 100,000 catalogue items changed today, rescoring all 100,000 wastes most of the run. Track what actually changed and rescore only that.

No plan for staleness. A precomputed table is a snapshot. It needs the same honest thinking about "how old is too old" as any other cache. See cache invalidation for RAG for that discussion, applied to a related problem.

Try it yourself

Increase CATALOG_SIZE to 50,000 and rerun. Watch whether the "x faster" number grows, shrinks, or stays about the same. Think about why, given what is doing the extra work in each version.

What to learn next

Researcher — Mathematics and papers.

When precomputation dominates on-demand cost

Let $n$ be the number of items to score, $c_{\text{batch}}$ the marginal cost of scoring one item inside a large batch, and $c_{\text{single}}$ the cost of scoring one item alone (including fixed per-call overhead — Python dispatch, any network hop to a model service). Precomputing wins outright whenever the items are reused across many requests, since the one-time cost $n \cdot c_{\text{batch}}$ is amortised over every subsequent lookup, each costing $O(1)$.

The batching gap itself, $c_{\text{single}} - c_{\text{batch}}$, is dominated by fixed overhead per call — memory allocation, framework dispatch, in a networked setting a full round trip — which does not scale with model size. This is the same effect exploited by dynamic batching at serve time. Precomputation is the extreme case: the "batch" is the entire known input space, computed once rather than assembled from concurrent live requests.

The staleness–cost frontier

Precomputing trades freshness for cost and latency. Formally, if the true best answer for an item can change over time, a precomputed table has an inherent staleness $\tau$ equal to the time since it was last refreshed. The design question is choosing a refresh cadence such that the business cost of staleness at $\tau$ stays below the compute cost of refreshing more often — a trade-off with no universal answer, since both sides depend entirely on the specific product.

Lambda architecture (Marz, 2011, popularised through Big Data) formalises the general pattern this lesson is a special case of. A slow, comprehensive batch layer recomputes the full precomputed view periodically. A fast, narrow speed layer covers the gap for anything too recent for the last batch run — matching exactly the "precomputed table plus live fallback" design recommended above.

Coverage versus completeness

A precomputed table's usefulness is bounded by its coverage: the fraction of real incoming requests it can actually answer from the table, rather than falling through to a live path. Coverage below roughly 90% for high-traffic keys usually means most of the precomputation's cost is not being recovered in saved live calls. Measure this directly against real production traffic logs, rather than assuming it.

Reading

  • Marz and Warren, Big Data: Principles and Best Practices of Scalable Realtime Data Systems, Manning, 2015 (the Lambda Architecture chapters)

What to learn next