Webhooks Are Four Problems, Not One HTTP Call
Calling a customer's URL sounds trivial. It's really four problems: who may subscribe, proving the call came from you, surviving their outage, and not slowing down the request that triggered it.
If someone wants to know when a ticket closes, they have to ask. Poll GET /tickets/{id}, or poll a job status endpoint. A customer whose CRM should update on every close would poll every ticket, forever.
Webhooks turn that around. The customer gives you a URL once, and you call it when something happens.
It sounds like “make an HTTP call.” It is really four problems: who may subscribe, how the receiver knows the call came from you, what happens when their server is down, and how to do all of it without slowing down the request that triggered it.
A URL from a customer is an attack surface
Your server now makes requests to URLs that users type in.
Someone subscribes http://169.254.169.254/latest/meta-data/ (the cloud metadata service) or http://localhost:8080/admin, and your API fetches it from inside your network. That’s SSRF, server-side request forgery.
In production:
- Require
https. - Resolve the host and refuse private, loopback, and link-local addresses. Check again at send time, since DNS can change.
- Don’t follow redirects.
- Send from a network segment that can’t reach anything internal.
In memory means lost on restart
A webhook queued in a Channel is gone if the process restarts before it’s sent. A standard resilience handler’s retries span seconds, not hours. Real providers retry for a day or more (Stripe: up to three days).
That needs a durable queue, a stored “next attempt” time, and a new timestamp and signature for each attempt, since receivers reject old timestamps. And the gap between saving a change and announcing it still applies here: the record can change and the process die before the event is queued. The outbox pattern closes it.
Rotating a secret
The signature header can carry several signatures, space-separated. To rotate, sign with both the old and the new secret for a while, so receivers can switch whenever they’re ready. Then drop the old one. Receivers should accept any matching signature in the list.
Thin or fat events
A thin payload carries the ID and who did what. The receiver calls back for the rest, with its own credentials, so it always sees current data and nothing leaks to a URL that shouldn’t have it.
A fat event (the whole record) saves that call. It also goes stale, and it makes every field part of the webhook contract. Thin is the safer default.
Ordering, duplicates, and dead endpoints
Webhooks arrive at least once and in no guaranteed order: parallel deliveries plus retries. Tell receivers so. Dedupe on the webhook ID, and use the payload’s timestamp rather than arrival order.
An endpoint that has failed every delivery for days should be disabled, with an email to its owner, rather than retried forever.
The point
The HTTP call is the smallest part of a webhook. Validate the URL like an attacker chose it, sign every attempt, make delivery durable, and assume receivers will see duplicates out of order.