Idempotency Explained: Handling Duplicate API Requests

Quick Answer: Idempotency is a system property where making the same request multiple times produces the same result as making it once. By passing a unique idempotency key with each request, servers can safely ignore accidental retries—ensuring users aren't charged multiple times if they mash a submit button.

You know the exact scenario. You're buying something online, and you get a bit frustrated because the internet is being flaky. You hit the bright "Buy" button and it doesn't work. Then you hit it three more times. Eventually, you get a loading spinner and think, "Okay, I think I've bought the thing."

Why haven't you been charged four times? Obviously, if you got charged four times, that would be really annoying.

I often look at these everyday scenarios from a systems perspective. When building software that handles transactions, preventing these accidental double-charges is a major priority. If an application doesn't handle network retries gracefully, you end up with frustrated users. To solve this, we rely on a technique called idempotency.

How does idempotency work in distributed systems?

Idempotency works by attaching a unique identifier to a client request, which the backend server tracks. If the client retries the exact same request with the same identifier, the server recognizes it as a duplicate and simply returns the cached success response without re-processing the logic.

One way of describing this would be to imagine you're working in a warehouse and you are building sheds for people. The guy that actually builds the sheds is at the back of the warehouse. It's a very loud warehouse, so you can't really hear the replies going on.

Someone comes up to you and says, "I want a shed." So you shout to the guy at the back, "Hey, build a shed!" He starts building a shed. You don't hear a reply, though, because it's very loud. Five minutes later, you're anxiously going, "I'm not sure if he's building the shed." So you shout again, "Hey, build me a shed!" The guy hears you and thinks it's a new order. Now you've got two sheds being built.

This is exactly what happens when a web browser sends duplicate HTTP requests to an API over a flaky network without idempotency.

What is an idempotency key and why is it required?

An idempotency key is a unique string generated by the client and sent alongside a request. It is required to give the server a reliable way to differentiate between a genuinely new request and an accidental retry of an existing request.

So how do you solve the loud warehouse problem? You come up with a system where you pass along a specific code.

Now, you shout: "Hey, build me a shed, code 123!" The guy at the back acknowledges, "Building a shed, code 123." Five minutes later, you get anxious because you haven't heard a reply. You shout again, "Build me a shed, code 123!" The builder immediately realizes this is a duplicate. He replies, "Yeah, code 123, I'm already building you the shed."

This is kind of what happens when your website communicates with a backend system. Your browser comes up with this little code. When you hit that buy button, it sends the code. The server receives your request, and it might receive that same code four times. The first time, it starts processing the payment. When it receives the three further requests with the same code, it knows it's already working on it. That's how you avoid paying four times when you mash the button.

How do you implement idempotency keys in an API?

To implement idempotency keys, you require clients to include an idempotency identifier in their requests. The server intercepts this identifier, checks a fast data store to see if the key exists, and either processes the request or returns the previously saved response.

Let's say your team is building a microservice handling payments. You generally want to handle idempotency checks at the edge or middleware layer before hitting your core business logic. Here is the standard step-by-step flow:

• Extract the key: Read the idempotency code from the incoming request payload or HTTP headers.
• Check the cache: Query your database or fast cache (like Redis) for this specific key.
• Handle in-progress requests: If the key exists and is marked as "processing," return a conflict or a custom "retry later" response to the client.
• Handle completed requests: If the key exists and is marked as "completed," return the exact status and body from the original successful response.
• Process new requests: If the key doesn't exist, save it to the cache as "processing," execute the actual payment logic, update the key's state to "completed," and return the result to the user.

By enforcing this workflow, the server guarantees that even if a browser sends four identical requests over a flaky connection, the underlying business logic only triggers on the very first attempt.

Frequently Asked Questions

Which HTTP methods are naturally idempotent?

GET, PUT, DELETE, HEAD, and OPTIONS are defined as naturally idempotent by the HTTP specification. Making these requests multiple times should not change the server state beyond the initial request. POST is not inherently idempotent, which is why manual idempotency keys are mandatory for sensitive endpoints like payments.

How long should a server store an idempotency key?

It depends on your business context, but a common standard is 24 hours. The goal of an idempotency key is to catch immediate network retries or user button-mashing, not to maintain a permanent ledger of every request ever made.

Can I use the user ID or order ID as an idempotency key?

You should avoid this. A user might legitimately want to place two identical orders back-to-back on the same day. Instead, generate a unique identifier (like a UUID) on the client side specifically for the individual form submission or button click.