"Webhook" sounds like a small word for something that quietly runs a good chunk of modern automation. A payment succeeds, a form gets submitted, a deploy finishes: somewhere a system fires an HTTP request at a URL and hopes it lands. Most of the time it does. The interesting engineering happens in the rest of the time, and that's what this note is actually about. Not the demo. The Tuesday afternoon where three things go wrong at once.
Idempotency Is The Whole Game
A retry is a second chance for a message to arrive. It is also a second chance for your system to do the same thing twice: charge a card twice, create two identical orders, send the same shipping notification three times because earlier responses got lost in transit. The fix isn't cleverer retry logic. It's making the receiving side able to shrug off a duplicate.
That means giving every meaningful event an identifier, and checking that identifier before you act on it. If you've already processed event id X, processing it again should be a no-op, not a repeat performance. It's the single most skipped step in real integrations, because it doesn't show up as a bug until traffic doubles and duplicates start arriving close enough together to race each other.
Idempotency doesn't have to be complicated: a table recording which event ids you've already handled, checked before any side effect runs, covers most cases. "Delivered" and "processed exactly once" are different claims, and only one is under your control.
Retries Deserve A Backoff Plan, Not Blind Faith
When a webhook fails to deliver, the honest question is why. A receiver that's down for ten seconds during a deploy needs a different response than a receiver that's structurally rejecting every payload you send it. Retrying immediately and repeatedly just adds load to a system that's already struggling to answer.
Backoff solves this by spacing retries out: try again soon after the first failure, then wait longer each time after that. Add a little randomness to the wait time, usually called jitter, so a thousand failed deliveries don't all retry in the same instant and create a second outage on top of the first. Set a ceiling. At some point, a message has failed enough times that retrying again isn't optimism, it's denial.
What that ceiling triggers matters as much as the backoff curve itself, which is the whole reason dead-letter handling deserves its own conversation and not just a footnote.
Dead Letters Are A Record, Not A Failure
A dead-letter queue is where a message goes after it has failed enough times that retrying automatically stops making sense. Teams without one have a worse system in disguise: the message just disappears. The only sign anything went wrong is a support ticket days later asking why an order never got confirmed.
Treat the dead-letter queue as a working surface, not a graveyard. Someone should be able to look at what's sitting in it, see why each message failed, and decide: replay it now that the downstream issue is fixed, patch the payload and resend it, or acknowledge it's genuinely unrecoverable and close it out on purpose, rather than by accident.
The queue itself should be boring infrastructure. The judgment about what to do with what lands in it is the part that still needs a human.
Verify The Signature Before You Trust The Payload
An endpoint that accepts webhooks is a public URL that does something when it receives a POST request. Anyone who finds that URL can send it a payload shaped like the real thing. If you act on whatever arrives without checking where it actually came from, you've built an API that anyone can call while wearing your legitimate sender's clothes.
Signature verification fixes this. The sender signs each payload with a shared secret, and you recompute that signature on your end before doing anything with the request. If it doesn't match, you reject it. Pair this with a timestamp or nonce check so a captured, valid payload can't simply be replayed later and accepted as new.
This step gets skipped more often than it should, usually because everything works fine without it in testing. Nobody attacks your staging environment. Production is a different audience.
Ordering Is A Promise Most Systems Don't Actually Keep
It's tempting to assume webhooks arrive in the order they were sent. Retries break that assumption immediately: if event two succeeds on the first try and event one needed three attempts, event two shows up first. Add multiple delivery workers running in parallel and you can get genuine out-of-order delivery, not just delayed delivery.
Design for this instead of hoping around it. Include a timestamp or sequence number in the payload itself and let your receiver decide whether an incoming event is newer than the state it already has, instead of trusting arrival order. For anything where the final state actually depends on sequence, like a status field that should only move forward, check the incoming event's position against what you've already recorded before applying it.
Get this wrong and the failure is quiet: everything appears to process successfully, and the data is just occasionally, subtly wrong.
When The Receiving End Is Down
Your endpoint will go down. A deploy, a database failover, a traffic spike unrelated to webhooks at all: at some point a sender will try to deliver and get nothing back. The question is what the sender does next, and if you're the one sending webhooks out to other people's systems, that answer is yours to design.
A sender with no retry logic just loses the event. A sender with unlimited immediate retries can turn a brief blip into a thundering herd the moment your endpoint comes back up. The middle path is the backoff and ceiling approach from earlier, paired with a dead-letter queue for anything that exhausts its attempts while you were unreachable.
If you're on the receiving end, make your endpoint fail fast and cheap when it's overloaded, so senders get a clean signal to back off instead of hanging.
Monitoring For What Didn't Happen
Error rate dashboards catch the failures that announce themselves: a spike in 500s, a queue depth climbing, a retry count going up. They're much worse at catching the failure that looks like nothing: an integration that quietly stopped sending events three hours ago, with zero errors logged anywhere, because nothing failed. Nothing happened. That's the entire problem.
The fix is monitoring for absence, not just errors. Track the expected rate of events for a given integration and alert when it drops well below normal, not just when it spikes. Watch dead-letter queue depth as its own signal, since a slow climb there often means something upstream changed shape before anyone noticed a hard failure. Treat prolonged silence from a webhook as its own incident category, separate from an outright error.
The engineers who get paged the least aren't the ones with zero failures. They're the ones whose monitoring can tell the difference between quiet and fine.
| Delivery Pattern | Reliability | Complexity To Build | Latency |
|---|---|---|---|
| Fire And Forget | Low | Low | Low |
| Retry With Backoff | Medium | Medium | Low to Medium |
| Queue Backed Delivery | High | Medium to High | Medium |
| Queue Backed With Dead Letter Handling | High | High | Medium |
Do I need a message queue to have reliable webhooks, or is retry logic enough?
Retry logic with backoff and idempotency will get a small integration a long way on its own. A queue earns its complexity once you have enough volume, or enough downstream steps, that you need to decouple receiving an event from finishing it. If a single slow step can hold up everything behind it, that's usually the signal you've outgrown retries alone.
How long should I keep retrying a failed webhook delivery before giving up?
There's no universal number, and anyone who gives you one hasn't run enough of these in production. The honest approach is picking a ceiling based on how stale an event can get before it's useless, then handing it to a dead-letter queue instead of retrying forever. A payment confirmation and an analytics ping have very different tolerances for staleness.
What's the simplest way to check that a webhook actually came from who it claims?
Verify the signature the sender attaches to the payload against a secret you both share, computed the same way on both ends, before trusting the payload's contents. If there's no signature to check, treat the source as unverified and be conservative about what you let it trigger, especially anything with a side effect that costs money or reaches a real person.
None of this is exotic engineering. It's a handful of decisions, made in advance, about what happens when something goes wrong instead of after. If you'd rather get the occasional field note than dig it out of your own postmortems, that's what we send. Get on the list.