Caching Explained

A cache is a small, fast store sitting in front of a large, slow one, keeping whatever's currently "hot" close at hand so most requests never have to reach the slow store at all. Cache Hit Ratio Calculator measures how well that's working — this post is about the mechanism underneath that number: what happens when the fast store fills up, what happens when the slow store changes underneath it, and the failure mode that catches teams who've only ever thought about the happy path.

Every layer of a real system is doing the same trade

The pattern repeats at every scale a request passes through, each layer trading capacity for speed: CPU L1/L2/L3 caches (kilobytes to megabytes, nanoseconds) sit in front of RAM (gigabytes, tens of nanoseconds), which sits in front of local disk (terabytes, microseconds to milliseconds), which — for a web request — sits behind a browser cache, a CDN edge cache, and an application-level cache (Redis, Memcached) in front of whatever actually computes the answer. None of these are conceptually different tools; they're the identical trade-off applied at a different point in the same request's path.

Eviction: what happens when the cache is full

A cache is, by definition, smaller than what it's caching — so something has to give when it's full and a new entry needs room. Which entry gets evicted is a genuine design decision with real trade-offs:

• LRU (Least Recently Used) — evict whatever hasn't been touched in the longest time. Assumes recent access predicts near-future access (temporal locality), a good bet for most real traffic. Implemented efficiently with a doubly linked list (tracking access order) plus a hash map (O(1) lookup by key) — see Array vs Linked List for exactly why that pairing is the standard answer, and How Hash Maps Work for the hash map half of it.
• LFU (Least Frequently Used) — evict whatever's been accessed the fewest times, ever. A different bet: some items stay consistently popular regardless of how recently they were touched. Costs more to track (a count per entry, not just a position) and, without a decay mechanism, an old item that was briefly viral can block eviction long after it stopped mattering.
• FIFO — evict whatever's been in the cache longest, full stop, ignoring access pattern entirely. Trivial to implement, and often measurably worse than LRU in practice — but "often worse" isn't "always worse," and the implementation simplicity is a real advantage when the workload doesn't reward the extra bookkeeping.

A concrete LRU trace, capacity 3, makes the policy concrete rather than abstract:

Step 4 is the detail worth noticing: accessing A again doesn't just return it — it moves A to the most-recently-used end, which is exactly why B, not A, gets evicted one step later even though A entered the cache first.

Write policy: what happens when the underlying data changes

Eviction handles reads; writes need their own decision about when the cache and the source of truth actually agree:

• Write-through — every write goes to the cache and the backing store together, synchronously. Always consistent, at the cost of every write paying the slow store's latency.
• Write-back (write-behind) — a write lands in the cache immediately and is marked dirty; the backing store gets updated later, asynchronously. Fast writes, at a real risk: a crash before that flush loses the write entirely.
• Write-around — a write goes straight to the backing store, skipping the cache entirely. Avoids filling the cache with data that was written once and may never be read again — at the cost of that same data being a guaranteed miss the first time it's actually read.

Invalidation: the actually hard part

"There are only two hard things in Computer Science: cache invalidation and naming things," often attributed to Phil Karlton, undersells it slightly — cache invalidation is hard specifically because a cache, by design, is a copy that can silently drift from the truth the moment the original changes. Two real strategies, with a real trade-off between them:

• TTL (time-to-live) expiration — every cached entry gets a lifespan; once it expires, the next request re-fetches it. Simple to reason about and requires no coordination with whatever wrote the change — at the cost of a guaranteed staleness window up to the full TTL, no matter how fast the actual write happens.
• Explicit invalidation — the write path actively tells every cache holding that data to drop or update it. No staleness window at all when it works — but it only works if the writer genuinely knows about every cache that might be holding a copy, which gets considerably harder the moment there's more than one cache layer or more than one service that can write the same data.

Cache stampede: the failure mode that only shows up under load

A single popular key expires. In the instant before it's repopulated, every concurrent request for that key misses at once — and every one of them independently goes to regenerate it, hitting the backing store with a spike of duplicate work exactly when it's least prepared for it. This is a cache stampede (or thundering herd), and it's specifically a high-traffic problem — it doesn't show up in testing with one request at a time, only once real concurrent load exists. Real mitigations: a lock so only the first miss regenerates the value while the rest wait for it, jittering TTLs so hot keys don't all expire in the same instant, or stale-while-revalidate (a real, standardized HTTP caching directive, RFC 5861) — serve the stale value immediately while regenerating it in the background, trading a moment of staleness for zero requests ever waiting on a cold cache.

Try it yourself

Cache Hit Ratio Calculator turns the outcome of everything above into the one number that actually tells you whether it's working. Throughput Calculator covers what happens on the other side of a cache miss — how much capacity the system behind it actually needs.

FAQ

Is a CDN just a cache?

Yes, specifically — a geographically distributed cache for HTTP responses, sitting between users and an origin server so a request can be served from a nearby edge location instead of crossing the network to origin every time. Same mechanism as every other layer in this post, applied at the network edge.

LRU or LFU — which is actually better?

Neither, universally — it depends on which assumption matches the real access pattern. LRU wins when recent access predicts near-future access, which describes most general web traffic. LFU wins when some items are consistently hot regardless of how recently they were touched, which describes things like reference data or a fixed popular catalog. Guessing wrong just means picking a policy whose assumption doesn't hold for the actual workload — not a bug, a mismatch.

Does a bigger cache always mean a better hit rate?

Usually, up to a point — beyond the point where the cache already holds everything that's genuinely hot, more space just holds colder and colder data for a shrinking return. Cache Hit Ratio Calculator covers what ranges are actually worth expecting for different cache types.