Kafka

Your message queue delivers an event twice. Your consumer processes it twice. The order ships twice, the email sends twice, the payment charges twice. At-least-once delivery is a guarantee — not a bug. Here's how to build idempotent consumers that handle duplicate events safely.

Order created at 10:00. Order cancelled at 10:01. Your consumer processes them in reverse — cancellation arrives first, then creation "succeeds." The order is now in an invalid state. Event ordering bugs are subtle, expensive, and entirely avoidable.

Your queue has 50 million unprocessed messages. Consumers are processing 1,000/second. New messages arrive at 5,000/second. The backlog will never drain. Here's how queue backlogs form, why they're dangerous, and the patterns to prevent and recover from them.