Webhook retries done right: backoff, jitter, and circuit breakers
A customer's receiver starts returning 503s. Your webhook sender notices, and retries. Immediately. So do the other 40,000 deliveries queued for that same endpoint. The receiver was already overloaded — which is why it returned 503 in the first place — and now it is getting hit harder than under normal load. It never recovers. This is a retry storm, and a naive retry loop causes it rather than preventing it.
Getting webhook retries right is mostly about restraint: backing off when things fail, spreading retries out so they don't synchronize into a thundering herd, giving up eventually instead of looping forever, and cutting off an endpoint that is clearly down so you stop wasting attempts on it. Here is how to build each piece, with the exact formulas and SQL from Dropless, an open-source webhook gateway written in Rust.
Why retry at all: at-least-once webhook delivery
Networks drop packets, receivers restart mid-request, load balancers return spurious 502s. If you give up after the first failure, you lose events that the receiver genuinely needed. So a webhook gateway commits to at-least-once delivery: keep trying until the receiver acknowledges with a 2xx, or until you've exhausted a sane retry budget.
"At-least-once" is the honest contract. Exactly-once is not achievable over an unreliable network — a receiver can process a request and then have its 200 response lost on the way back, at which point the sender, having seen no ack, retries. The fix is not to chase exactly-once; it is to make retries safe to receive twice. Every request carries an immutable Idempotency-Key (we reuse the delivery's id), so the receiver dedupes on it and a double-delivery is a no-op on their side. Build your retry logic assuming duplicates will happen, because they will.
At-least-once only holds if an acknowledged event is never silently lost before it is even retryable. In Dropless an event and all of its fan-out deliveries are written in a single Postgres transaction, and the HTTP API returns 2xx only after that commit — commit-then-2xx. A crash commits everything or nothing, so there is no acknowledged-but-vanished event to retry in the first place. Retries handle delivery failures; the transaction handles ingestion failures.
Exponential backoff: the formula
The first rule of retrying is: wait longer each time. A receiver that just failed needs room to breathe. Exponential backoff doubles the wait after each attempt:
delay(attempt) = base * 2^(attempt - 1)
With base = 2s, that gives 2s, 4s, 8s, 16s, 32s, and so on. Two refinements make it production-safe:
- A ceiling.
2^15is over nine hours; you don't want a single delivery's interval to grow without bound. Cap each interval at amax_delay(we use one hour). - Saturating math.
2^attemptoverflows a 64-bit integer fast. Clamp the exponent and saturate the multiply so a high attempt count can never panic or wrap to a tiny delay.
Here is the ceiling computation from Dropless, in Rust:
/// Capped ceiling delay for a 1-based attempt, before jitter.
fn ceiling(&self, attempt: i32) -> Duration {
// 2^(attempt-1) * base, saturating, then capped at max_delay.
let exp = attempt.saturating_sub(1).clamp(0, 30) as u32;
let factor = 1u64.checked_shl(exp).unwrap_or(u64::MAX);
let millis = (self.base.as_millis() as u64).saturating_mul(factor);
Duration::from_millis(millis).min(self.max_delay)
}
Why exponential backoff jitter matters
Backoff alone has a subtle failure mode. Suppose an endpoint goes down for 30 seconds and 5,000 deliveries fail at roughly the same instant. They all compute the same backoff — say 8 seconds — and all retry at the same instant 8 seconds later. You've just rebuilt the retry storm, only on a delay. The deliveries are synchronized, and they will stay synchronized through every subsequent backoff step, hammering the recovering endpoint in waves.
Jitter breaks the synchronization by randomizing each delay. The variant worth using is full jitter (popularized by AWS): instead of waiting exactly the computed delay, wait a uniform random amount between zero and that delay.
delay = random_between(0, base * 2^(attempt - 1)) // full jitter
This spreads the 5,000 retries evenly across the whole window instead of bunching them at the end. The mean wait is half the ceiling, and — more importantly — the variance smooths the load into a flat trickle the recovering endpoint can absorb. In Rust it is a single line on top of the ceiling above:
/// Backoff for the given (1-based) attempt with full jitter applied.
pub fn backoff(&self, attempt: i32) -> Duration {
let ceil_ms = self.ceiling(attempt).as_millis() as u64;
if ceil_ms == 0 {
return Duration::ZERO;
}
let jittered = rand::thread_rng().gen_range(0..=ceil_ms);
Duration::from_millis(jittered)
}
People sometimes worry full jitter "wastes" the lower half of the window by retrying too early. In practice that earliness is the point: it desynchronizes the herd, which matters far more than squeezing a few seconds out of any single retry.
Max attempts and the dead letter queue
Retrying forever is its own bug. An endpoint whose URL was deleted, or whose owner forgot to renew TLS, will never succeed — looping on it burns connections and clogs the queue behind healthier traffic. So every delivery has a max attempts budget (we default to 16). Once a delivery exhausts it, the sender stops retrying and moves the delivery to a dead letter queue: a terminal dead state recording the last error and attempt count.
The dead letter queue is not a graveyard; it's a tray of failures an operator can inspect and act on. Once the receiver is fixed, you replay the dead-lettered deliveries with one click and they re-enter the normal pipeline. Crucially, dead-lettering is a deliberate decision, not a silent drop — the event is preserved, just parked.
One thing worth getting right: distinguish transient from permanent failures. A non-2xx response or a connection timeout is transient — schedule a retry. But a payload that won't serialize, or a signing error, will fail identically on every attempt. Retrying a poison row just reclaims it forever without progress, so dead-letter those immediately rather than spending the whole attempt budget on a guaranteed failure.
Per-endpoint circuit breaker for webhooks
Backoff and jitter pace the retries for a single delivery. But when an endpoint is hard-down, you want to stop attempting all of its deliveries, not pace each one independently. That's the job of a circuit breaker, and it must be per-endpoint — one tenant's broken receiver shouldn't slow deliveries to everyone else.
The breaker is a three-state machine stored on the endpoint row (circuit_state, circuit_open_until, consecutive_failures):
- Closed — normal. Deliveries flow. Each consecutive failure increments a counter; a success resets it to zero.
- Open — once consecutive failures cross a threshold (default 5), the breaker trips. For a cooldown window (default 30s) every delivery to this endpoint is deferred immediately, without an HTTP attempt. This is the load relief: the struggling receiver gets a quiet window instead of a barrage.
- Half-open — when the cooldown elapses, allow exactly one probe request through. If it succeeds, close the circuit and resume. If it fails, re-open for another cooldown.
The "exactly one probe" part is where naive implementations leak. If the cooldown elapses and the whole backlog is released at once to test the waters, you've recreated the storm you were avoiding. The fix is to make the probe a contended claim: the first worker to atomically flip the endpoint to half_open wins the probe; everyone else defers. Keeping the decision logic side-effect-free makes it trivially unit-testable, while the atomic claim lives in one guarded SQL UPDATE.
Scheduling webhook retries in Postgres
Backoff is only a number until something acts on it. In Dropless, Postgres is the only runtime dependency — it is both the source of truth and the queue, with no Redis or Kafka — so a scheduled retry is just a future timestamp in a column. Recording a failure appends an immutable attempt row and stamps the next eligible time, all guarded by the worker's lease so a delivery another worker has since re-claimed is never clobbered:
UPDATE deliveries
SET status = 'failed',
attempt_count = $2,
next_attempt_at = $3, -- now() + jittered backoff
last_error = $4,
locked_by = NULL,
locked_until = NULL,
updated_at = now()
WHERE id = $1 AND locked_by = $5 AND locked_until > now();
Workers then claim due work with a single statement that doubles as crash recovery — it picks up both freshly-due retries and rows whose owner died mid-attempt (their lease has lapsed):
UPDATE deliveries AS d
SET status = 'in_progress',
locked_by = $1,
locked_until = now() + ($2::int * interval '1 second'),
updated_at = now()
WHERE d.id IN (
SELECT id FROM deliveries
WHERE (status IN ('pending', 'failed') AND next_attempt_at <= now())
OR (status = 'in_progress' AND locked_until < now()) -- reclaim crashed
ORDER BY next_attempt_at
FOR UPDATE SKIP LOCKED
LIMIT $3
)
RETURNING d.*;
FOR UPDATE SKIP LOCKED lets many workers pull disjoint batches without coordination, and a poll interval (backed by LISTEN/NOTIFY to wake idle workers) guarantees timed retries fire even when nothing new is enqueued. Because the lease expiry reclaims abandoned rows, a worker killed mid-delivery costs you at most a duplicate (which the idempotency key absorbs), never a lost event. This is enforced by a real kill -9 test in CI that hard-kills the dispatcher mid-delivery and asserts drop = 0.
Putting it together
A correct retry path is a small stack of independent decisions: retry because the network is unreliable; back off so a struggling receiver can recover; add full jitter so retries don't synchronize; cap attempts and dead-letter the rest so the queue stays clean; and trip a per-endpoint circuit breaker so a hard-down endpoint stops consuming attempts entirely. None of these is exotic — but skip any one and you get the retry storm you were trying to prevent.
If you'd rather not build this yourself, Dropless implements all of it — exponential backoff with full jitter, max-attempts plus a replayable dead letter queue, and a per-endpoint circuit breaker — on Postgres alone. It is open source under AGPL-3.0 (Docker: ghcr.io/r4nd0menryu/dropless), and the code shown above is the actual scheduling and breaker logic. There is also a hosted version if you'd rather skip the ops.