Idempotency in Distributed Systems

In distributed systems, messages get lost, responses timeout, and clients retry. If your API creates a new order every time the same request is retried, you will end up with duplicate orders, double charges, and angry customers. Idempotency ensures that performing the same operation multiple times produces the same result as performing it once.

---

The Problem

1. Client sends: POST /orders {items: [...]}
2. Server creates order #123, charges $99.99
3. Response lost in transit (network timeout)
4. Client retries: POST /orders {items: [...]}
5. Server creates order #124, charges $99.99 again

Result: Customer charged twice. Two duplicate orders.

---

Idempotency Key Pattern

1. Client generates unique idempotency key
2. Client sends: POST /orders {items: [...]} 
   Header: Idempotency-Key: idem_abc123
3. Server creates order #123, stores: idem_abc123 → order #123
4. Response lost in transit
5. Client retries with same key: Idempotency-Key: idem_abc123
6. Server finds key in store → returns original response

Result: One order, one charge, same response both times.

Implementation

# Server-side idempotency middleware
class IdempotencyMiddleware:
    def __init__(self, redis_client):
        self.redis = redis_client
    
    async def process_request(self, request, handler):
        key = request.headers.get('Idempotency-Key')
        if not key:
            return await handler(request)  # Non-idempotent request
        
        cache_key = f"idempotency:{request.path}:{key}"
        
        # Check for existing response
        cached = await self.redis.get(cache_key)
        if cached:
            return json.loads(cached)  # Return original response
        
        # Acquire lock to prevent concurrent processing
        lock = await self.redis.set(
            f"lock:{cache_key}", "processing",
            nx=True, ex=60  # 60-second lock timeout
        )
        if not lock:
            return Response(status=409, body="Request in progress")
        
        try:
            response = await handler(request)
            
            # Cache response for 24 hours
            await self.redis.set(
                cache_key, json.dumps(response.body),
                ex=86400  # 24 hours
            )
            return response
        finally:
            await self.redis.delete(f"lock:{cache_key}")

Database-Level Idempotency

-- Unique constraint prevents duplicates
CREATE TABLE orders (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    idempotency_key VARCHAR(255) UNIQUE NOT NULL,
    customer_id UUID NOT NULL,
    total_amount DECIMAL(10, 2) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Insert only if idempotency key does not exist
INSERT INTO orders (idempotency_key, customer_id, total_amount)
VALUES ('idem_abc123', 'cust_456', 99.99)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING *;

---

Naturally Idempotent Operations

Idempotent by nature:
  GET /orders/123          → Always returns same order
  PUT /orders/123 {data}   → Sets state to exact value
  DELETE /orders/123       → Order is deleted (or already was)

NOT idempotent by nature:
  POST /orders {data}      → Creates new order each time
  PATCH /orders/123 {qty: qty + 1}  → Increments each time
  POST /payments/charge    → Charges each time

Making non-idempotent operations safe:
  POST /orders → Add idempotency key
  PATCH with increment → Use absolute values: {qty: 5} not {qty: +1}
  POST /payments → Idempotency key + payment reference

---

Message Queue Deduplication

# Kafka consumer with deduplication
class IdempotentConsumer:
    def __init__(self, db):
        self.db = db
    
    async def process_message(self, message):
        message_id = message.headers.get('message-id')
        
        # Check if already processed
        if await self.db.execute(
            "SELECT 1 FROM processed_messages WHERE message_id = $1",
            message_id
        ):
            logger.info(f"Duplicate message {message_id}, skipping")
            return
        
        # Process within transaction
        async with self.db.transaction():
            await self.handle(message)
            await self.db.execute(
                "INSERT INTO processed_messages (message_id, processed_at) VALUES ($1, NOW())",
                message_id
            )

---

Anti-Patterns

Anti-Pattern	Consequence	Fix
POST without idempotency key	Duplicate resources on retry	Require idempotency key for mutations
Idempotency in memory only	Lost on server restart	Persist in database or Redis
No TTL on idempotency records	Unbounded storage growth	24-48 hour TTL, configurable
Client-generated sequential IDs	Collisions between clients	UUIDv4 or ULID for idempotency keys
Increment operations (qty + 1)	Double-increment on retry	Use absolute values (qty = 5)

Idempotency is not optional in distributed systems. It is the foundation of reliable, retry-safe APIs and event-driven architectures.