Proving crash-safety with a real kill -9 test

Your delivery pipeline accepts an event, returns 200 OK, and the producer moves on. Three milliseconds later the dispatcher process gets OOM-killed by the kernel. Question: does that event still get delivered?

Almost every webhook gateway and job queue answers "yes, of course" in its README. Very few have ever actually run the experiment. The gap between "we use a durable store" and "we have killed the process mid-write and watched it recover" is exactly where acknowledged-but-lost events live. The only way to close it is to perform the crash on purpose, in CI, on every commit.

This post walks through a real kill -9 test for a webhook delivery pipeline: it publishes events, sends SIGKILL to the dispatcher while deliveries are genuinely in flight, restarts a fresh process, and asserts two things — zero dropped events, and correct lease reclaim of the rows the dead worker was holding. It is the test that backs the durability claim in Dropless, a self-hostable webhook gateway written in Rust whose only runtime dependency is Postgres.

Why a kill -9 test catches what unit tests miss

A graceful shutdown is not a crash. When you send SIGTERM and let the process drain its in-flight work, you are testing the happy path — the one that almost never fails in production. SIGKILL (signal 9) is different: it is uncatchable and unblockable. The process does not get to flush buffers, finish an HTTP request, write a "failed" status, or release a lock. It simply stops existing between two CPU instructions.

That is the realistic failure. The kernel OOM killer, a container runtime reaping a pod that blew its memory limit, a node losing power, a hypervisor pausing a VM into oblivion — all of them look like kill -9 from inside your process. If your durability story depends on a destructor running or a "mark this as failed on shutdown" handler firing, it is not a durability story. It is a hope.

The architectural precondition that makes the test meaningful is commit-then-2xx. In Dropless, an inbound event and every fan-out delivery row it produces are written in a single Postgres transaction, and the HTTP API returns 2xx only after that transaction commits. A crash either commits the whole transaction or none of it. There is no window where the caller got a 200 but the rows are not durably on disk. If you ack before you commit, no downstream test can save you — you have already lied to the producer.

The Postgres claim query that makes crash recovery possible

Recovery is not a separate code path you bolt on for crashes. In a Postgres-backed queue it falls out of the claim query itself. Dropless uses a single store — Postgres is both the source of truth and the queue, with no Redis or Kafka in the picture — and workers claim due deliveries with FOR UPDATE SKIP LOCKED plus a visibility-timeout lease stored in two columns: locked_by and locked_until.

UPDATE deliveries AS d
SET status = 'in_progress',
    locked_by = $1,
    locked_until = now() + ($2::int * interval '1 second'),
    updated_at = now()
WHERE d.id IN (
    SELECT id FROM deliveries
    WHERE (status IN ('pending', 'failed') AND next_attempt_at <= now())
       OR (status = 'in_progress' AND locked_until < now())
    ORDER BY next_attempt_at
    FOR UPDATE SKIP LOCKED
    LIMIT $3
)
RETURNING d.*;

Read the WHERE clause carefully, because the second branch is the entire crash-recovery story. A row is claimable if it is pending or failed and due, or if it is already in_progress but its locked_until lease has expired. When a worker dies under kill -9, the rows it held stay frozen at status = 'in_progress' with a locked_until in the future — the dead worker never got to release them. They look stuck. But the lease is time-bounded: once now() passes locked_until, the second predicate matches and any live worker re-claims them. No reaper job, no liveness gossip, no leader election. The reclaim is implicit in the same query that does normal claiming.

SKIP LOCKED is what lets multiple workers run this concurrently without coordination: each claim takes row locks and skips rows another transaction already holds, so two workers never grab the same delivery in the same instant. The result is at-least-once delivery. We do not pretend it is exactly-once — a worker can POST a webhook, get a 200 back, and crash before writing succeeded, so the row gets redelivered after the lease lapses. That is precisely why every request carries an immutable Idempotency-Key and a Svix-compatible webhook-id header for receivers to dedupe on. The test's job is to prove the "at-least" half is airtight: the drop count must be exactly zero, while duplicates are expected and allowed.

The kill -9 durability test, end to end

The test is a shell script wired into CI. It runs against a real Postgres and real OS processes — no mocking the database, no aborting tasks in-process. The skeleton:

# 1. Migrate, create an isolated tenant + endpoint pointing at a mock receiver.
"$BIN" migrate
"$BIN" create-api-key  --tenant "$TENANT" --key "$KEY"
"$BIN" create-endpoint --tenant "$TENANT" --url "http://127.0.0.1:$MPORT/" --secret "..."

# 2. Start node A with a SHORT lease (LOCK_SECS=2) so reclaim is fast.
LOCK_SECS=2 "$BIN" serve --role=all --bind "127.0.0.1:$PORT" &
SRV_PID=$!

# 3. Publish N=200 events; record each returned delivery_id as "sent".
for i in $(seq 1 "$N"); do
  curl -s -XPOST "http://127.0.0.1:$PORT/v1/messages" \
    -H "Authorization: Bearer $KEY" \
    -d "{\"event_type\":\"load.test\",\"payload\":{\"n\":$i}}" \
    | jq -r '.delivery_ids[0]' >> "$SENT"
done

# 4. Let deliveries get IN FLIGHT, then SIGKILL the node. No graceful drain.
sleep 0.7
kill -9 "$SRV_PID"

# 5. Wait out the 2s lease so the dead worker's rows become reclaimable, restart.
sleep 3
start_node   # node B, a fresh process

# 6. Drain, then compare the SET sent against the SET received.
missing="$(comm -23 <(sort -u "$SENT") <(sort -u "$RECV") | grep -c .)"
[[ "$missing" -ne 0 ]] && { echo "FAIL: $missing dropped"; exit 1; }
echo "PASS: 0 dropped across kill -9"

The mock receiver deliberately sleeps ~40ms per request. That latency is not incidental — it guarantees deliveries are still mid-flight at the moment of the kill, so the dead worker really is holding leased rows. The final assertion is a set comparison: comm -23 of sent IDs minus received IDs must be empty. Duplicates are allowed and reported; drops fail the build.

Guarding against a vacuous pass in chaos testing

The subtle failure mode of any chaos test is that it passes without testing anything. If all 200 deliveries happen to complete before the kill, the script still prints PASS — but it never exercised reclaim. The crash was a no-op. A green check then means nothing.

The companion Rust integration test closes this hole explicitly. After aborting the workers, it queries the database and refuses to continue unless something was genuinely caught in the act:

let in_flight: i64 = sqlx::query_scalar::<_, i64>(
    "SELECT count(*) FROM deliveries \
     WHERE tenant_id = $1 AND status = 'in_progress'",
)
.bind(&tenant)
.fetch_one(&pool)
.await
.unwrap();

assert!(
    in_flight > 0,
    "crash should leave deliveries in_progress to exercise reclaim; found {in_flight}"
);

If a regression ever makes deliveries drain too fast — or breaks the reclaim predicate — this assertion fails loudly instead of letting a meaningless pass ship. After it confirms in-flight rows exist, it waits out the lease, restarts the worker pool, and asserts both that every webhook-id arrived at least once and that the database shows zero deliveries left in any non-succeeded state. Two independent witnesses — what the receiver saw, and what the table says — have to agree before the test goes green.

Why crash-safety belongs in CI, not in a runbook

Durability is not a feature you implement once; it is an invariant you can break with a one-line refactor. Reorder an ack before a commit, widen a lease to "never expires," swap a transaction for two statements, change the claim predicate to skip the in_progress branch "for performance" — any of these silently turns an at-least-once system into a lossy one. Nothing in a type checker or a normal unit test will catch it. The only thing that catches it is killing a process and counting.

So the crash proof runs on every pull request, as a required gate, alongside a second gate worth mentioning: the entire workspace must build and unit-test without a database connection. Dropless uses only sqlx's runtime-checked query API — never the compile-time query! macros — so cargo build needs no live Postgres, and CI enforces that by building with DATABASE_URL unset. The reliability job then spins up a real Postgres service container and runs both the in-process no-loss proof and the out-of-process kill -9 script against it. A delivery-loss regression cannot merge green.

If you take one thing from this: the README claim "we never lose events" is worth exactly as much as the test that kills the process and proves it. Write the test. Make it fail when you weaken the invariant. Put it in the pipeline.

Dropless implements everything described here — commit-then-2xx, lease-based reclaim with FOR UPDATE SKIP LOCKED, and the kill -9 proof as a CI gate. It is open source under AGPL-3.0 at github.com/r4nd0mENRYU/dropless (Docker image: ghcr.io/r4nd0menryu/dropless), or run it as a hosted service. The full test lives in scripts/kill9_test.sh and crates/dropless/tests/no_loss.rs if you want to read or borrow it.