The newsletter goes out at 9:00. At 9:00:01, four hundred readers land on the front page, and the cached copy expired at 8:59:58. Every worker misses at the same instant, every one of them starts rendering the same page, and the database spends its morning answering one question four hundred times. The caching recipe calls this a thundering herd. For most values it’s harmless. When the value is slow to build (a front page that takes 400 ms, a report that queries a warehouse), there are two ways to make sure readers never see the gap, and only one worker does the work.
Stale-while-revalidate
Store the value with its own freshness deadline inside, and give the entry a longer hard expiry. A reader that finds the entry fresh returns it. One that finds it stale returns it anyway, and the first of them to win a claim starts a refresh in the background. Readers never wait, and the herd is one worker.
import tempfile
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from typing import Any
from diskcache_oxide import Cache
now = [1_000_000.0] # a fake clock; in a real app use time.time()
cache = Cache(tempfile.mkdtemp(), clock=lambda: now[0])
refresher = ThreadPoolExecutor(max_workers=2) # one per process, for background refreshes
loads: list[float] = []
def load_front_page() -> dict:
time.sleep(0.05) # stands in for 400 ms of rendering
loads.append(now[0])
return {'rendered_at': now[0]}
def store(key: str, value: Any, fresh_for: float, grace: float) -> Any:
cache.set(key, {'value': value, 'fresh_until': now[0] + fresh_for}, expire=fresh_for + grace)
return value
def refresh(key: str, fresh_for: float, grace: float) -> None:
try:
store(key, load_front_page(), fresh_for, grace)
finally:
cache.delete(f'refresh:{key}')
def front_page(fresh_for: float = 60, grace: float = 300) -> dict:
key = 'page:front'
entry = cache.get(key)
if entry is None: # a real miss: nothing to serve, compute inline
return store(key, load_front_page(), fresh_for, grace)
if now[0] >= entry['fresh_until'] and cache.add(f'refresh:{key}', 1, expire=30):
refresher.submit(refresh, key, fresh_for, grace) # stale, and this worker won the refresh
return entry['value'] # fresh or stale, the reader gets it now
front_page() # the first request renders the page
now[0] += 61 # a minute later it's stale
results = []
start = threading.Barrier(8)
def request() -> None:
start.wait()
results.append(front_page()['rendered_at'])
threads = [threading.Thread(target=request) for _ in range(8)]
for t in threads:
t.start()
for t in threads:
t.join()
refresher.shutdown(wait=True) # in the example only: let the refresh land
print(f'{len(results)} readers got the stale page from t={results[0]:.0f}, {len(loads)} renders in total')
print(f"the next reader gets the fresh page from t={front_page()['rendered_at']:.0f}")
8 readers got the stale page from t=1000000, 2 renders in total
the next reader gets the fresh page from t=1000061
The three numbers to choose:
fresh_for is how old a value may get before a refresh starts. It plays the role expire plays in a plain cache.grace is how much longer the stale value stays available. If no refresh lands within it (the refresher crashed, the source was down), the entry expires and the next reader computes inline, herd and all. Make it generous: minutes, not seconds.- The claim’s expire (30 s here) bounds how long a crashed refresher blocks the next one. Make it longer than a refresh ever takes.
In an asyncio app, start the refresh with asyncio.create_task(asyncio.to_thread(refresh, ...)) instead of a thread pool; see Using the cache from asyncio. With durable=True, a failed add() returns False, so nobody refreshes until the cache is healthy again, which is the right outcome.
Refreshing ahead from a job
When the hot keys are known in advance (the front page, the top ten episodes, the pricing table), it’s simpler to never let them go stale: a scheduled job, claimed by one worker, recomputes them every few minutes and stores each with an expire of at least twice the interval.
HOT_PAGES = ['front', 'episodes', 'pricing']
def refresh_hot_pages() -> None: # runs every 5 minutes, on one worker
if not coordination.add('job:refresh-hot-pages', os.getpid(), expire=4 * 60):
return
for name in HOT_PAGES:
cache.set(f'page:{name}', render_page(name), expire=15 * 60)
Readers do a plain get() and never miss; a missed run or two costs nothing because the entries outlive several intervals. The cost is that the job renders pages nobody may be asking for, which is fine for a handful of keys and wrong for a long tail. Stale-while-revalidate handles the tail, since it only refreshes what’s being read.
A memoize decorator with this built in is planned (see the roadmap).