Message Queues Explained

"Exactly-once delivery" is a phrase that shows up in a lot of marketing copy and is, in the general case, impossible — not hard, impossible. A network can't distinguish "the message was lost" from "the acknowledgment was lost," so any system built on top of an unreliable network has to pick a side: risk sending a message twice, or risk not sending it at all. Everything a message queue actually does — acking, redelivery, idempotency, dead-letter queues — is machinery built around that one unavoidable fact.

What a queue is actually for

A message queue decouples a producer from a consumer in two specific ways: time (the consumer doesn't have to be running, or even exist yet, when the producer sends a message — the broker holds it) and rate (the producer doesn't have to wait for the consumer to keep up; the broker buffers the difference). A direct synchronous call — an HTTP request straight to the downstream service — has neither property: both sides have to be up at the same instant, and the caller is blocked exactly as long as the callee takes. A queue turns that into two independent problems: getting the message accepted, and eventually processing it.

Delivery guarantees: what "reliable" actually means

Three tiers, and the difference between them is entirely about what happens when something fails partway through:

• At-most-once — fire and forget. The producer sends, the broker doesn't track whether it was ever processed, and a consumer crash between receiving a message and finishing work on it loses that message permanently. Simple and fast, appropriate only when losing an occasional message is genuinely fine (metrics, non-critical logs).
• At-least-once — the broker keeps the message until the consumer explicitly confirms it's done, and redelivers it if that confirmation doesn't arrive in time. This is the practical default for most real systems (SQS, RabbitMQ with manual acks) because it never silently drops a message — the cost is that the same message can be delivered more than once, which makes consumer idempotency mandatory, not optional.
• Exactly-once — the effect of processing a message happens exactly one time, no duplicates, no losses. As stated above, this can't be built from network delivery alone. What Kafka calls "exactly-once semantics" (since 0.11, 2017) is specifically an idempotent producer (deduplicating retried sends via a producer ID and sequence number) plus transactional writes for a Kafka-to-Kafka read-process-write pipeline — a real, useful guarantee, but scoped to Kafka talking to Kafka, not a blanket promise about whatever a consumer does with a message once it's in hand.

The mechanic behind at-least-once: visibility timeout

This is the concrete machinery, using the model SQS makes explicit (RabbitMQ's ack/nack does the equivalent job with a manual timeout instead of an automatic one): when a consumer receives a message, the broker doesn't delete it — it makes that message invisible to other consumers for a configurable window (SQS defaults to 30 seconds) and waits for an explicit delete/ack. If the ack arrives in time, the message is gone for good. If it doesn't — the consumer crashed, hung, or is just slow — the visibility timeout expires and the message becomes available again, to be picked up and processed a second time.

Queue vs topic: two genuinely different delivery models

These get conflated constantly, and the difference isn't branding — it's a different answer to "how many consumers see each message."

Real systems implement this differently, and the differences matter operationally:

• RabbitMQ separates the two concerns explicitly: an exchange routes messages, and a queue holds them for delivery. A fanout exchange bound to multiple queues gives topic-style fan-out; a single queue with multiple consumers gives queue-style load balancing (round robin by default) — both are available, and which one you get depends entirely on the exchange type and binding, not the queue itself.
• Kafka gets both models from one structure at once: a topic is split into partitions, and consumers are organized into consumer groups. Within one group, each partition is assigned to exactly one consumer (queue-style — the group as a whole processes each message once). But every separate consumer group subscribed to the topic gets its own full copy of the stream (topic-style fan-out across groups). Also unlike a traditional queue, Kafka doesn't delete a message on consumption — it's retained per the topic's retention policy, and each consumer group tracks its own offset, so a new consumer group can replay the entire history.
• SQS is queue-only — no native fan-out to multiple independent consumer groups. The standard pattern for topic-style behavior is SNS (a separate pub/sub service) publishing to multiple subscribed SQS queues, each with its own consumers — composing two AWS services to get what Kafka does natively with one.

Ordering: guaranteed less often than it's assumed

A plain SQS standard queue makes no ordering promise at all — messages can be delivered out of the order they were sent, a direct consequence of the same distributed, redundant-by-design storage that makes it durable. An SQS FIFO queue guarantees order, but only within a message group ID — messages in different groups have no ordering relationship to each other. Kafka works the same way at the partition level: order is only guaranteed within a single partition, which is exactly why messages that must stay ordered relative to each other (all events for one user, say) need to share a partition key that hashes to the same partition. Across partitions — like across SQS message groups — there's no ordering guarantee, by design, because that's what allows parallelism in the first place.

Dead-letter queues: what stops a bad message from looping forever

At-least-once delivery plus a consumer that reliably crashes on one specific malformed message creates an infinite loop: redeliver, crash, redeliver, crash — a "poison message" that never gets acked and never goes away, potentially blocking every message behind it in the queue. A dead-letter queue (DLQ) is the fix: after a message has been redelivered past a configured retry count without being acked, the broker routes it to a separate queue instead of retrying again, unblocking the main queue and leaving the failed message somewhere a human (or a separate remediation process) can actually inspect it.

Backpressure: queue depth is the signal, not the problem

A growing queue depth means the producer is outpacing the consumer — the same capacity question Load Balancers Explained covers for synchronous request routing, just measured differently. Little's Law (L = λW: items in the system equals arrival rate times average time each item spends in the system) applies directly — if consumers can't process messages as fast as they arrive, queue depth grows without bound, and average message age grows with it. The fix is never "a bigger queue"; it's more consumer throughput (more consumers, since a queue naturally load-balances across them) or a lower arrival rate.

Try it yourself

Throughput Calculator applies Little's Law directly to this — model whether a given number of consumers at a given processing latency can actually keep up with an arrival rate before queue depth becomes the problem in production instead of on paper. Distributed ID Explorer is the practical companion to the idempotency requirement above: at-least-once delivery means a consumer needs a reliable way to recognize "I've already processed this exact message," and a sortable, collision-resistant ID (ULID, KSUID, or a Snowflake-style ID) embedded in the message payload is the standard way to build that dedup key.

Real systems, compared

FAQ

Is Kafka a message queue?

Loosely, yes, but precisely it's a distributed commit log that a queue can be built on top of. The distinction that actually matters: a traditional queue (RabbitMQ, SQS) deletes a message once it's consumed, while Kafka retains it for the topic's retention period regardless of consumption, which is what makes replay and multiple independent consumer groups possible in the first place.

Do I need a dead-letter queue if my consumer never throws exceptions?

Almost certainly still yes — a DLQ isn't insurance against bugs you know about, it's insurance against the failure mode you didn't anticipate: a malformed message from an upstream system, a downstream dependency that's down long enough to exceed every retry, a schema change nobody coordinated. Without one, that failure mode is an infinite redelivery loop instead of a message sitting somewhere visible.

If I use at-least-once delivery, how do I actually make my consumer idempotent?

Track which message IDs have already been fully processed — a unique ID per message (generated once, at creation, not regenerated on redelivery) checked against a dedup store before doing the work, or a database write that's naturally idempotent on its own (an upsert keyed by that same ID rather than an insert). Either way, the ID has to be stable across redeliveries of the same logical message, which is exactly why it needs to be assigned once by the producer rather than derived from receive time.