Horizontal vs Vertical Scaling treated the load balancer as a black box — it "spreads load" and "routes around failures." Neither of those is automatic; both are specific, real mechanisms a load balancer has to implement, and which one it uses changes how the system actually behaves under real traffic.
Layer 4 vs Layer 7: how much of the request it actually reads
Layer 4 (transport layer) load balancing looks only at IP and TCP/UDP info — source and destination address and port — and forwards accordingly, without parsing anything above that. Fast, low overhead, and protocol-agnostic — it'll balance any TCP traffic, not just HTTP. Layer 7 (application layer) load balancing terminates the actual protocol (HTTP, gRPC) and can read the URL path, headers, and cookies — enabling routing decisions an L4 balancer structurally can't make at all, like sending /api/* to one pool and /static/* to another, or pinning a specific client to a specific backend by reading a cookie. The cost is real too: parsing the actual protocol is more work than forwarding packets blind. Many real deployments use both in sequence — a fast L4 balancer at the edge distributing raw connections, an L7 balancer behind it making the smart routing decisions.
Which backend gets the next request
- Round robin — cycle through backends in order. Simple, and only fair if every request costs roughly the same amount of work — a mix of cheap and expensive requests breaks that assumption immediately.
- Weighted round robin — the same cycle, but backends with a higher assigned weight get proportionally more turns — the fix for a pool of differently-sized instances.
- Least connections — send the next request to whichever backend currently has the fewest active connections. Better than round robin whenever request cost varies, since it reacts to actual current load instead of assuming everything costs the same.
- IP hash — deterministically map a client's IP to the same backend every time. The L4-level way to get sticky behavior without needing to read a cookie — at the cost of breaking if that client's IP changes mid-session, which mobile networks do more often than it's convenient to assume.
- Power of two random choices — pick two backends at random, send the request to whichever of those two has fewer active connections. First analyzed by Michael Mitzenmacher (1996 dissertation, formalized 2001): checking just two candidates instead of every backend gives exponentially better load distribution than pure random selection, at a fraction of least-connections' overhead of tracking every backend continuously. Nginx and HAProxy both ship this as a real, named algorithm — it's not a theoretical curiosity, it's in production load balancers today.
How "routes around failures" actually happens
A load balancer only avoids a dead backend if something tells it the backend is dead — two real mechanisms, often used together:
- Active health checks — the balancer independently pings a health endpoint on every backend at a fixed interval; a backend that fails enough consecutive checks gets pulled from rotation until it starts passing again.
- Passive health checks — no independent pinging at all; the balancer watches real traffic, and a backend that starts erroring or timing out on actual requests gets marked unhealthy from that alone.
These aren't universally bundled the way it's easy to assume — open-source nginx supports passive health checks out of the box, but active health checks are an NGINX Plus (paid) feature specifically; HAProxy, by contrast, supports both directly. Which one a given deployment actually has determines how fast — and how reliably — it notices a backend has died at all.
Sticky sessions: the tension with balancing evenly at all
Spreading requests evenly and keeping one client's requests on one specific backend are directly in tension — the second one exists because some session data historically lived only in one backend's memory, so a client's follow-up request had to land back on that same machine or lose its session entirely. Two mechanisms do this: an L7 balancer setting and reading a cookie that pins a client to a backend, or L4 IP-hash routing as a coarser, cookie-free approximation of the same idea.
The more resilient fix is avoiding the need for stickiness at all — keep session state in a shared store every backend can read (Redis is the standard choice; see Caching Explained for the write-policy trade-offs that apply to it directly) instead of in any one instance's own memory. That's also precisely what makes a backend genuinely stateless in the first place — a Horizontal vs Vertical Scaling prerequisite this post has just made concrete.
DNS round robin: load balancing before any load balancer is involved
A domain can resolve to multiple IP addresses (multiple A records), and a client's OS picks one — a crude form of load balancing that happens at the DNS layer, before a request ever reaches a real load balancer. It has no concept of backend health at all: a dead server's IP stays in the DNS answer, and handed out to new clients, until the record's TTL expires and it gets removed — which can be minutes, depending on how the TTL was configured. A dedicated load balancer's active/passive health checks react in seconds; DNS round robin reacts on whatever timescale the TTL happens to allow.
Try it yourself
Throughput Calculator models the exact assumption a load balancer is responsible for making true — that N instances behind it genuinely add up to N times one instance's capacity, which only holds if load is actually spread evenly across them.
FAQ
Should I use L4 or L7?
Depends on what the routing decision needs to know. If every backend is interchangeable and the only question is which one is least busy, L4 is faster and simpler. The moment routing needs to depend on the URL path, a header, or a cookie, that information only exists at L7 — an L4 balancer structurally cannot see it.
Can round robin ever perform worse than random selection?
Yes, in a specific real pattern: if requests arrive with any cyclical structure that happens to correlate with the round-robin cycle length, or if backends periodically slow down in a pattern round robin's fixed order keeps re-hitting at the wrong moment, its determinism can create synchronized load spikes that a randomized algorithm — power of two choices included — naturally avoids.
What happens if every backend fails its health check at once?
Genuinely implementation-specific, and worth confirming for whatever's actually in use rather than assuming — common behaviors include returning an error to the client (502/503) or, in some configurations, "failing open" and routing to a backend anyway as a last resort rather than serving zero traffic. Neither is obviously correct in the abstract; it depends on whether a wrong answer or no answer is the worse outcome for that specific service.