Decouple work with message queues

A message queue (SQS, RabbitMQ) or log (Kafka) sits between services that produce work and services that consume it. The producer returns as soon as the message is stored, so slow work - sending emails, transcoding video, updating feeds - happens asynchronously. Queues also level load: a burst piles up in the queue instead of overwhelming consumers, which drain it at their own pace.

• At-most-once: a message may be lost but is never redelivered.
• At-least-once: a message is redelivered until acknowledged, so it may arrive more than once. This is the common default.
• Exactly-once: very hard end to end. In practice you get effectively-once by combining at-least-once delivery with idempotent consumers.

An operation is idempotent if applying it twice has the same effect as once. Consumers achieve this by recording processed message ids (or using an idempotency key) and skipping duplicates.

A worked example: deliveries arrive as pay-1 for $50, pay-2 for $20, then pay-1 again. A consumer that records processed ids skips the repeat, so the balance is $70, not $120.

Other queue patterns come up often. Work queues give each message to one consumer; pub/sub delivers each message to every subscriber. Kafka keeps order only within a partition, so you pick a partition key (such as user id) for events that must stay ordered. A dead-letter queue collects messages that keep failing so one bad message does not block the rest. Watch consumer lag: if producers outpace consumers for long, the backlog grows without bound.