DROPLESSBlogDocsGitHub

At-least-once webhooks and why idempotency keys matter

2026-06-24 · 7 min read

The duplicate charge that wasn't a bug

A customer gets billed twice for one order. You pull the logs and find two identical checkout.completed webhooks, milliseconds apart, same payload, same everything. Your sender didn't malfunction. Your receiver didn't malfunction. The network in between hiccuped at exactly the wrong moment, and a correctly-built delivery system did precisely what it was designed to do: it retried.

This is the defining problem of webhook delivery, and it has nothing to do with bugs. It comes from a property of distributed systems that no amount of careful code can engineer away. Understanding it is the difference between a receiver that double-charges customers and one that shrugs off duplicates.

The exactly-once delivery myth

"Exactly-once delivery" sounds like the obvious goal, and it's what most people assume a good webhook system provides. It is, over a network, impossible. Not hard — impossible.

Walk through a single delivery. The sender POSTs your event. Your receiver processes it, commits to its database, and sends back 200 OK. Then, before that response reaches the sender, the TCP connection drops. The sender is now stuck with a genuinely unanswerable question:

From the sender's side, a lost acknowledgement and a lost request are indistinguishable. The bytes that would tell them apart are exactly the bytes that went missing. This is the Two Generals' Problem, and it has a proof attached: no fixed number of messages over a lossy channel can give both parties certainty. You have to pick a failure mode.

Pick "never retry on uncertainty" and you get at-most-once: no duplicates, but you silently drop events whenever an ack is lost. For billing, order fulfillment, or audit trails, that's unacceptable. Pick "always retry on uncertainty" and you get at-least-once: never lose an acknowledged event, but accept that some will arrive more than once. Every serious webhook system chooses the second. The duplicates are the price of never losing data, and they are a price you can actually pay — on the receiver, with an idempotency key.

Where webhook duplicates actually come from

It's worth being concrete about the sources, because they're not exotic:

None of these are correctable at the sender. They're inherent to delivering over an unreliable channel with a finite timeout. So the contract Dropless offers is honest and specific: at-least-once. An event and all of its fan-out deliveries are written in one Postgres transaction, and the API returns 2xx only after that transaction commits — commit-then-2xx. A crash commits everything or nothing; there is no acknowledged-but-lost event. Once we've acked your event, it will be delivered — possibly more than once, never zero times.

What at-least-once delivery means for your receiver

At-least-once delivery moves one responsibility onto the receiver, and only one: make processing idempotent. Handling the same event twice must produce the same result as handling it once. The mechanism that makes this practical is a stable identifier attached to each logical event, carried on every retry of that event.

Dropless sends headers derived from the delivery id, all immutable across retries:

That stability is the whole point. A key that changed per attempt would be useless for deduplication. Because it doesn't change, the receiver can record "I've seen this key" and reject the second arrival cheaply.

A receiver dedupe pattern that survives concurrency

The naive version — SELECT to check if you've seen the key, then INSERT — has a race. Two duplicate deliveries can both pass the SELECT before either INSERTs, and you've double-processed. The fix is to let the database arbitrate with a uniqueness constraint and do the insert and the business write in one transaction:

-- Once: a table whose primary key is the idempotency key.
CREATE TABLE processed_webhooks (
    idempotency_key text PRIMARY KEY,
    processed_at    timestamptz NOT NULL DEFAULT now()
);
-- Per request, inside ONE transaction:
BEGIN;

-- Claim the key. If a concurrent or earlier delivery already
-- claimed it, this inserts zero rows.
INSERT INTO processed_webhooks (idempotency_key)
VALUES ($1)               -- the Idempotency-Key header
ON CONFLICT (idempotency_key) DO NOTHING;

-- Only do the real work if WE won the claim (1 row inserted).
-- If 0 rows: it's a duplicate. COMMIT and return 200 — the
-- first delivery already did the work, so this is a no-op success.

-- ... your side effects here: fulfil order, apply credit, etc.

COMMIT;

The application logic around it, in pseudo-Rust (sqlx):

let mut tx = pool.begin().await?;

let claimed = sqlx::query(
    "INSERT INTO processed_webhooks (idempotency_key) \
     VALUES ($1) ON CONFLICT DO NOTHING",
)
.bind(&idempotency_key)
.execute(&mut *tx)
.await?
.rows_affected();

if claimed == 0 {
    // Already processed by an earlier delivery of this same key.
    tx.rollback().await?;
    return Ok(StatusCode::OK); // ack the duplicate; do nothing
}

apply_business_effect(&mut tx, &event).await?; // the real work
tx.commit().await?;                            // dedupe + effect atomic
Ok(StatusCode::OK)

Two properties make this correct. First, the unique constraint makes the "have I seen this?" check atomic — the database, not your application, resolves the race. Second, the dedupe insert and the side effect commit together, so you can never end up in the half-state where the key is recorded but the work didn't happen (or vice versa). If the transaction aborts, nothing is recorded, and the sender's retry gets a clean shot at it. That's at-least-once doing its job.

Verify the HMAC signature before you trust the key

An idempotency key is only meaningful if the request is authentic — otherwise anyone can replay or forge one. Verify the webhook-signature first, with a constant-time comparison. Recompute the HMAC over the signed string "{webhook-id}.{webhook-timestamp}.{raw_body}", base64-encode it, and compare it against the candidate after the v1, prefix in the header (the header can hold several tokens; match against any of them). Use the raw request bytes, not a re-serialized JSON object: re-encoding can reorder keys or change whitespace and break the HMAC. Reject timestamps that are too far off to blunt replay attacks. Only after the signature checks out should you let the key drive your dedupe table.

This is exactly what the Dropless inbound gateway does for sources it ingests: POST /ingest/{slug} verifies Stripe, GitHub, or generic-HMAC signatures with constant-time comparison, persists the raw body before anything else, dedupes on the provider's event id, then bridges into the outbound pipeline. Same discipline, just on the receiving edge.

Why this beats chasing exactly-once

You could try to push the complexity back onto the sender — distributed transactions, two-phase commit across a network you don't control, consensus protocols. People have spent careers on this. The pragmatic, battle-tested answer the whole industry converged on is simpler and more robust: the sender guarantees at-least-once, and the receiver dedupes on a stable key. Effective exactly-once processing emerges from at-least-once delivery plus an idempotent handler. The network stays honest, and your customer gets charged once.

Dropless is an open-source (AGPL-3.0) webhook delivery gateway written in Rust, with Postgres as its only runtime dependency — source of truth and queue in one. It implements exactly the contract described here: commit-then-2xx so acknowledged events are never lost, at-least-once delivery with lease-based crash recovery, and a stable Idempotency-Key (plus Svix-compatible signatures) on every delivery so your receiver can dedupe with the pattern above. A CI "kill -9" test kills the dispatcher mid-delivery and asserts zero drops. Source and docs: github.com/r4nd0mENRYU/dropless. Prefer hosted? See dropless.dev.

Dropless implements everything above — open source (AGPL-3.0): github.com/r4nd0mENRYU/dropless · ← more posts