Backend Development › Queues & Async Processing

Idempotent Consumer

A consumer that handles duplicate messages safely.

Also known as: idempotent consumers, idempotent message handling, deduplicating consumer, idempotent receiver

An idempotent consumer processes the same message more than once and ends up in the same state as if it had processed it once. Because delivery is usually at-least-once, duplicates will arrive, and a consumer that isn’t idempotent corrupts data when they do (double charges, duplicate emails, inflated counters).

Techniques

1. Make the operation naturally idempotent. Express the result, not the change:

set_order_status(order_id, "shipped")        # repeating is harmless
add_to_balance(account_id, 100)              # repeating double-counts

2. Track processed message IDs. Record each handled message in the same transaction as the effect:

def handle(msg):
    with db.transaction():
        inserted = db.execute(
            "INSERT INTO processed_messages (message_id) VALUES (%s) ON CONFLICT DO NOTHING", (msg.id,)
        )
        if inserted == 0:
            return                                    # already handled: skip
        apply_business_change(msg)                    # same transaction: both happen or neither

The unique constraint on message_id makes the check race-safe (unique constraints). This is sometimes formalized as the inbox pattern.

3. Use unique constraints or upserts on the business data itself (UNIQUE (order_id, event_type)).

4. Conditional updates / versions: apply a change only if the state is what you expect, or if the message’s version is newer than the stored one (optimistic locking).

5. Pass idempotency keys to external calls (payment providers) so their side is protected too (idempotency key).

Pitfalls

• Checking, then acting, then recording in separate steps. A crash between steps leaves a gap. Put the record and the effect in one transaction, where you can.
• Side effects outside your database (emails, HTTP calls) can’t be rolled back. Use idempotency keys, or accept and design for rare duplicates.
• Where IDs come from: they must be stable across redeliveries, not generated by the consumer. Producers set them (a UUID in the message), ideally derived from the business event.
• Unbounded dedupe tables: expire old IDs after a window longer than the maximum redelivery period.
• Out-of-order messages: duplicates plus reordering can apply an old update after a new one. Include a version or timestamp, and ignore stale ones.

“Exactly-once processing” in practice usually means at-least-once delivery plus an idempotent consumer (exactly-once).