Imagine building a checkout flow. A user clicks "Pay Now." Your backend server forwards the charge request to a payment gateway like Stripe. Stripe takes the money. But right before your server can send a success response to the user, the network connection drops. The user sees a spinning wheel, gets impatient, and clicks "Pay Now" again.
If your system is not built to handle this, the customer gets charged twice. They get angry. Your support team gets flooded with tickets. You end up paying refund fees and dealing with chargebacks.
This is the classic double-spend problem in distributed systems. You cannot trust the network. Connections drop, servers restart, and clients retry. To make your payment system safe, you must design it to be idempotent. An idempotent operation is one that you can run multiple times with the exact same result as running it once.
The Anatomy of a Request Failure
When a client talks to your payment service, the interaction involves three parties: the client, your backend, and the downstream payment gateway.
[ Client ] - - - - ( Request ) - - - - > [ Your Backend ] - - - - ( Charge ) - - - - > [ Payment Gateway ]
^ | |
| v v
[ Client ] < - - - -( Timeout ) - - - - - [ Backend ] < - - - -( Success ) - - - - - - [ Gateway ]
If the connection breaks between the client and your backend after the gateway has processed the payment, the client is left in the dark. The client does not know if the payment succeeded or failed.
The only safe option for the client is to retry. But if your backend treats every incoming request as a brand-new transaction, you will charge the customer twice.
To prevent this, you need a way to identify that the second request is a retry of the first one.
The Idempotency Key Pattern
To identify retries, the client must generate a unique identifier for the transaction before sending the request. This is the idempotency key. The industry standard is to send this key in an HTTP header, usually named Idempotency-Key. The key is typically a UUID v4.
When your server receives a request, it follows a specific sequence:
- Look up the key in your database.
- If the key exists, check the status of the transaction. If it is finished, return the saved response. If it is still processing, return an error or tell the client to wait.
- If the key does not exist, save the key with a status of
processingand start the transaction. - Once the transaction completes, update the status and save the response body.
- Return the response to the client.
This sounds straightforward, but implementing this in a distributed system introduces race conditions and consistency issues.
Choosing Your Storage Layer
You need a storage engine to keep track of these keys. Your choice depends on your reliability requirements and how long you need to store the keys.
For most payment systems, you do not need to keep idempotency keys forever. Keeping them for 24 to 72 hours is usually enough. After a few days, the chance of a client retrying the exact same request is close to zero.
Redis as the Key Store
Redis is fast and supports automated expiry via Time-To-Live (TTL) settings. It can handle high write loads easily.
But Redis has a major drawback: durability. If you use a standard Redis configuration and the node crashes, you could lose recently saved keys. If a client retries during this window, you will double-charge the customer. You can mitigate this by using Redis Sentinel or clustering with persistence turned on, but it adds operational complexity.
Relational Database as the Key Store
If you cannot tolerate even a single double-charge, you should use your primary relational database, like PostgreSQL or MySQL.
You can create a dedicated table for idempotency keys:
CREATE TABLE idempotency_keys (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
key_value VARCHAR(255) UNIQUE NOT NULL,
request_hash VARCHAR(64) NOT NULL,
status VARCHAR(50) NOT NULL, - 'STARTED', 'COMPLETED', 'FAILED'
response_code INT,
response_body TEXT,
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);Using your primary database allows you to run the payment record insertion and the idempotency key check inside the same database transaction. This gives you strong consistency. You can use a background worker to delete rows older than three days to keep the table size manageable.
Handling Concurrent Requests (The Race Condition)
What happens if a user double-clicks the purchase button, and two identical requests hit your servers at the exact same millisecond?
If both requests check the database at the same time, both will see that the key does not exist. Both will proceed to charge the customer.
To prevent this, you must use a database lock or an atomic insert.
If you are using Redis, you can use the SET command with the NX (set if not exists) and PX (expiry time in milliseconds) options. This acts as an atomic lock.
SET idempotency:key_12345 "STARTED" NX PX 10000
If the command returns OK, your server has acquired the lock and can proceed. If it returns nil, another request is already processing this key. The second request must wait or fail.
If you are using PostgreSQL, you can rely on the UNIQUE constraint on the key_value column. When a request arrives, you attempt to insert the key with a status of STARTED.
INSERT INTO idempotency_keys (key_value, request_hash, status)
VALUES ('key_12345', 'hash_abc123', 'STARTED')
ON CONFLICT (key_value) DO NOTHING;After running this query, check if a row was actually inserted. If no row was inserted, it means the key already exists. You must query the database to find out the current status of that key.
The State Machine of a Key
An idempotency key moves through a simple state machine:
[ Key Created ] - -> ( STARTED ) - -> [ Process Payment ] - -> ( COMPLETED )
- -> ( FAILED )
When a request arrives and the key is in the STARTED state, it means the first request is still processing. You should not start a new transaction. Instead, return an HTTP 409 Conflict status code. You can also include a Retry-After header telling the client to try again in a few seconds.
When the key is in the COMPLETED state, read the saved response code and response body from the database and return them directly to the client. The client gets the exact same response as if the first request had succeeded.
If the key is in the FAILED state, your action depends on the type of failure. If the payment failed because the card was declined, this is a permanent failure. You should cache the declined response. Retrying with the same key should return the same decline message.
If the payment failed because of a transient error, like a database timeout on your end, you want to allow the client to retry. In this case, you can delete the idempotency key record or update its status to allow a retry.
Payload Validation and Fingerprinting
Never trust the client to use the idempotency key correctly. A buggy client might generate a single key and use it for two completely different transactions. For example, it might send a request to pay 10 for a book, and then send a second request to pay 100 for a phone using the same key.
If you only look at the key, your server will see the second request, find the cached response for the 10 book, and return success. The user gets the 100 phone but only pays `10.
To prevent this, you must validate the request payload. You do this by creating a cryptographic hash of the request body, using SHA-256, and storing it alongside the idempotency key.
When a retry arrives, compare the incoming request hash with the stored hash. If they do not match, the client is trying to reuse a key for a different payload. Return an HTTP 400 Bad Request status code with an error message explaining the mismatch.
Downstream Gateway Failures
Your server is often just a middleman. The actual money movement happens at the payment gateway.
What happens if your server calls Stripe, Stripe charges the card, but your server crashes before saving the COMPLETED state to your database?
The next time the client retries, your database will still show the state as STARTED or the record might be missing entirely if your database transaction rolled back. If you just run the payment again, you will double-charge the customer.
To solve this, you must pass your own idempotency key downstream to the payment gateway. Most modern payment processors support idempotency keys in their APIs.
By passing your key (or a key derived from it) to the gateway, you ensure that even if your server crashes, the gateway will not process the payment twice.
When your server recovers or receives the retry, it should query the gateway using the idempotency key to check the status of the transaction before attempting to charge the card again.
A Practical Implementation
Here is a simplified TypeScript implementation using Express and Redis to demonstrate the locking and validation flow.
import express, { Request, Response, NextFunction } from 'express';
import { createHash } from 'crypto';
import Redis from 'ioredis';
const app = express();
const redis = new Redis();
app.use(express.json());
interface IdempotencyRecord {
status: 'STARTED' | 'COMPLETED';
hash: string;
responseCode?: number;
responseBody?: string;
}
// Helper to generate a hash of the request body
function generateHash(body: any): string {
return createHash('sha256').update(JSON.stringify(body || {})).digest('hex');
}
app.post('/payments', async (req: Request, res: Response, next: NextFunction): Promise<void> => {
const key = req.headers['idempotency-key'];
if (!key || typeof key !== 'string') {
res.status(400).json({ error: 'Missing Idempotency-Key header' });
return;
}
const requestHash = generateHash(req.body);
const redisKey = `idempotency:`{key}`;
try {
// Attempt to acquire the lock and set status to STARTED
const recordStr = await redis.get(redisKey);
if (recordStr) {
const record: IdempotencyRecord = JSON.parse(recordStr);
// Validate that the request body matches the original request
if (record.hash !== requestHash) {
res.status(400).json({ error: 'Idempotency key reuse with different payload' });
return;
}
if (record.status === 'STARTED') {
res.status(409).json({ error: 'Request is already processing. Please retry later.' });
return;
}
if (record.status === 'COMPLETED') {
res.status(record.responseCode || 200).json(JSON.parse(record.responseBody || '{}'));
return;
}
}
// Lock the key for 10 seconds to prevent concurrent requests
const acquired = await redis.set(
redisKey,
JSON.stringify({ status: 'STARTED', hash: requestHash }),
'NX',
'EX',
10
);
if (!acquired) {
res.status(409).json({ error: 'Concurrent request detected' });
return;
}
// Process the payment with the downstream gateway
// Ensure you pass the key to the gateway too
const gatewayResponse = await mockPaymentGatewayCall(req.body, key);
const successResponse = {
transactionId: gatewayResponse.id,
status: 'success',
};
const completedRecord: IdempotencyRecord = {
status: 'COMPLETED',
hash: requestHash,
responseCode: 200,
responseBody: JSON.stringify(successResponse),
};
// Store the final response and set a longer TTL (e.g., 24 hours)
await redis.set(redisKey, JSON.stringify(completedRecord), 'EX', 86400);
res.status(200).json(successResponse);
} catch (error) {
// On failure, remove the lock so the client can retry
await redis.del(redisKey);
next(error);
}
});
async function mockPaymentGatewayCall(body: any, key: string) {
// Simulate network latency
await new Promise((resolve) => setTimeout(resolve, 500));
return { id: `tx_{Math.random().toString(36).substr(2, 9)}` };
}API Design Best Practices
When exposing idempotency to your API consumers, keep your interface clean and predictable.
Use standard HTTP response headers to inform the client about the status of the idempotency key. If the response is served from the cache, add a custom header like Idempotency-Cache: true. This helps clients debug their integration.
Ensure your documentation clearly states which endpoints support idempotency. Typically, POST and PATCH requests need it, while GET and DELETE requests do not, as they are naturally idempotent when designed correctly.
