The Valley of Webhooks

The 3rd time

I’ve built the aforesaid strategy 3 times now, astatine 3 different companies, for 3 different providers. It ne'er has a sanction and it ne'er appears connected a roadmap, but it ever goes the aforesaid way: the truth astir your ain customers lives successful personification else’s database. The users unrecorded successful an personality provider, the subscriptions successful Stripe, the bounces successful immoderate sends your email, and your merchandise needs that truth locally. So you subscribe to webhooks and support a copy.

The first time, I thought I was building an endpoint: 1 way that parses the JSON and updates a row, an day of work.

The day grew into a week. First came signature verification, because an unfastened endpoint that mutates your database is simply a hole. Then the dedup table, because deliveries get doubly and the docs cheerfully telephone this “at-least-once.” Then the handler sewage a buffer, because a membership.created sometimes shows up earlier the user.created it points to. Then the bootstrap importer, because webhooks only show you what happens after you subscribe, and it raced the unrecorded events, truthful it grew a locking scheme. And yet came the reconciliation cron: a occupation that crawls the provider’s database APIs astatine 3 a.m., diffs them against our tables, and softly fixes what disagrees.

I want to beryllium honorable astir what that cron is. It’s a written confession. It says: I do not spot the transcript I built, and I person nary measurement to cognize erstwhile it’s wrong, truthful I will re-derive it from scratch each night, forever.

The spot was gone for a reason. The drift ne'er announces itself; ours was recovered by a support ticket. A customer had cancelled months earlier and our database still said active; immoderate customer.subscription.deleted had evaporated betwixt Stripe and us, and thing anyplace was tin of noticing: not their dashboard, which showed the transportation arsenic retried and yet dropped, and not our logs, which cannot log a petition that ne'er arrived.

And that’s only the code. Every supplier besides brings its ain dashboard. Three providers, 3 webhook configuration pages, each pinch its ain thought of really endpoints are registered, which events exist, really trial and unrecorded environments are kept apart, and wherever the signing concealed lives. When thing breaks, debugging is simply a tour: their transportation log successful 1 tab, our logs successful another, and a 3rd tab for whichever dashboard I presently suspect. None of them look alike, and each of them person to beryllium checked.

By the 3rd clip I built this system, I had stopped pretending. I budgeted for the full stack up beforehand (signatures, dedup, buffering, bootstrap, cron) and location successful the mediate of penning my 3rd dedup table, I yet asked the mobility I should person asked the first time.

What precisely americium I reconstructing here?

Notifications aren’t data

I was reconstructing an ordered log. Every 1 of those integrations was an effort to move a watercourse of notifications backmost into the ordered, complete, existent history it came from.

And here’s the absurd part: that history exists. It has to, because it’s sitting wrong the provider; it’s really they render their dashboards, their arena pages, and their webhook replay tools. The supplier takes their ordered log, shreds it into individual HTTP POSTs, fires them astatine my endpoint complete a transmission that guarantees neither bid nor delivery, and past I reassemble the log connected my side. So does each different consumer, independently, each pinch their ain bugs.

It’s a jigsaw puzzle wherever the shaper had the original picture, trim it up, mailed maine the pieces 1 astatine a time, mislaid a fewer successful the post, mailed immoderate twice, and printed thing connected the box. And erstwhile my assembled puzzle doesn’t lucifer the original, their support squad asks me which pieces I’m missing. I don’t know, and that’s the full problem: thing announces a gap.

None of this is immoderate provider’s bug. Their webhooks activity precisely arsenic documented. The problem is what a webhook is: a notification, “something happened, here’s a POST astir it.” Notifications are a good measurement to trigger a broadside effect and a unspeakable measurement to transportation a dataset, and location on the measurement we started utilizing them for the 2nd point without noticing we’d changed jobs.

How did this go the norm?

Nobody decided this. The word “webhook” was coined by Jeff Lindsay successful 2007, and the early uses were genuinely bully fits: GitHub’s post-receive hooks kicking disconnected a CI build, aliases a costs arena pinging your server truthful it could email a receipt. The occupation was to do a point erstwhile a point happens, and for that a POST is perfect: fire-and-forget is good erstwhile forgetting is fine.

Webhooks dispersed because they were the cheapest point a supplier could vessel (one HTTP POST) and the cheapest point a user could person (you already had a web server, truthful you conscionable added a route). By the early 2010s, “we person webhooks” was a checkbox connected each API’s landing page, and the checkbox ne'er distinguished betwixt 2 very different jobs:

1. Trigger a broadside effect: nonstop the receipt, commencement the build, ping the channel.
2. Keep a transcript of the provider’s information correct: this customer deleted their costs method, truthful update it successful your DB too.

Job 1 is what webhooks were calved for. Job 2 is what I was doing each 3 times, and occupation 2 is the 1 wherever each spot webhooks deficiency (ordering, completeness, bootstrap, verifiability) is precisely the spot you need.

We picked the instrumentality that was lying connected the array successful 2007, and past we spent 15 years compensating.

The valley

There’s a conception successful evolutionary biology I can’t extremity reasoning about: the fittingness landscape. Peaks are bully designs, valleys are bad ones, and populations climb immoderate slope they hap to beryllium opinionated on. The trap is the local optimum: a mini elevation that’s amended than its contiguous surroundings, truthful improvement parks there, moreover erstwhile a overmuch higher highest exists crossed the valley. Getting to the higher highest intends crossing done designs that are temporarily worse, and improvement doesn’t do temporarily worse.

Webhooks-for-replication are a section optimum, and the impervious is the heap of workarounds connected the vale floor: signature schemes, dedup stores, idempotent handlers, retry queues pinch exponential backoff connected the supplier broadside and dead-letter queues down them, webhook logs pinch replay tooling because consumers support asking for replays, and my 3 a.m. cron.

The heap has an system connected apical of it. Svix exists truthful providers don’t person to build webhook delivery; Hookdeck exists truthful consumers don’t person to build webhook ingestion. AWS will waste you the vale arsenic managed services, pinch EventBridge to ingest your SaaS partners’ events, SQS to queue them, and Lambda to retry your handler, and you get to combine the pipeline yourself. And an full manufacture of connector platforms (Fivetran, Airbyte, each “unified API” startup) is, astatine bottom, pseudo-CDC: alteration information seizure reconstructed from webhooks and polled database APIs, 1 bespoke connector astatine a time, sold arsenic a product. Inside a database, capturing changes is simply a solved problem: it’s called replication, and it useful because there’s a log. Between companies, we rebuild it retired of doorbells.

My favourite workaround of them each is the section tunnel. Many providers vessel a CLI for illustration stripe perceive that opens a passageway to your laptop, because a webhook cannot scope localhost. Think astir what that is: a product, built and maintained by the provider, reinvented aggregate times, whose full intent is to activity astir the transportation guidance of their ain primitive. When aggregate providers each request to vessel a section passageway truthful developers tin develop, the primitive is answering the incorrect question.

None of this tooling is bad engineering; it’s fantabulous engineering. That’s what a section optimum looks like: truthful overmuch fantabulous engineering poured into the vale level that the vale becomes comfortable, and cipher looks up.

But immoderate providers person looked up. Stripe retains thirty days of events and exposes /v1/events, an ordered, listable log, and recommends reconciling against it. WorkOS ships an Events API, an ordered cursor-paginated log, and their ain docs urge it complete webhooks erstwhile information consistency matters. The log keeps escaping, and each flight mints its ain bespoke cursor semantics, its ain bootstrap story, nary measurement to verify a replica, and nary shared contract, but the guidance is unmistakable. This is convergent evolution: unrelated organisms, aforesaid biology pressure, aforesaid wing.

The log exists everywhere, but the statement doesn’t.

Could it beryllium better?

Before reaching for a caller design, it’s worthy asking what immoderate replacement would really person to provide. My 3 integrations propose the list: order, truthful changes tin beryllium applied without buffering; a measurement to commencement from nothing, truthful bootstrap isn’t a abstracted import racing the unrecorded events; deletes arsenic data, truthful absence stops being the nonaccomplishment mode; resumability, truthful my downtime is my problem alternatively of a data-loss event; and immoderate measurement to verify the result, truthful spot doesn’t decay into a 3 a.m. cron.

Measured against that list, the evident candidates travel up short. Polling the database APIs harder is the reconciliation cron promoted to a full strategy: it tin rebuild existent state, but it burns complaint limits discovering that mostly thing changed, it says thing astir order, and a deleted entity looks identical to an entity that ne'er existed. Managed delivery, whether that’s Svix connected the provider’s broadside aliases EventBridge and SQS connected mine, makes the pushes much reliable, but they are still pushes: still nary bootstrap, still nary verification, still notifications pretending to beryllium a dataset. That way hardens the vale level without climbing anywhere.

The 3rd campaigner is the 1 the providers support half-building connected their own: extremity pushing altogether, and fto the user publication the log itself.

Flip the arrow

So here’s the thought experiment. What if alternatively of the supplier telling america erstwhile location is caller information, we inquire the supplier what caller accusation it has for america since we past checked?

Suppose a supplier served 1 URL per collection, and that URL returned an ordered, cursor-addressed alteration log of full-state events. Ask without a cursor and you publication from the beginning, which is your bootstrap, pinch nary abstracted import and nary race. Ask pinch a cursor and you resume wherever you near off. Your full sync authorities is that cursor.

GET /feed/customers?cursor=01J9XQ4R Prefer: stream 200 OK Content-Type: application/x-ndjson {"cursor":"01J9XR2M","operation":"upsert","object":{"id":"cus_123","plan":"pro"}} {"cursor":"01J9XR2N","operation":"delete","object_id":"cus_099"}

Send Prefer: watercourse and the consequence ne'er ends: each alteration arrives arsenic it commits, complete a relationship you opened, utilizing the aforesaid API cardinal you usage for the normal REST endpoints. Leave it disconnected and you get a bounded page you tin canvass from a cron. It’s the aforesaid endpoint, the aforesaid events, the aforesaid cursors, and the aforesaid user code.

None of this is exotic; it’s a paginated GET. But locomotion backmost done my afternoon-that-grew and watch what it does to the stack:

• The dedup table is gone. Every arena carries the object’s afloat existent state, truthful applying 1 is simply a unsighted upsert keyed by id, and the aforesaid arena applied doubly produces the aforesaid result.
• The ordering buffer is gone, because the log is ordered.
• The bootstrap importer and its locking scheme are gone. A caller user sounds the aforesaid provender pinch nary cursor, replays the collection, and carries consecutive connected into unrecorded changes successful 1 request.
• The mislaid delete is impossible. A tombstone is an arena successful the log, and it sits location until I publication it. My cancelled customer cannot silently enactment active, because absence stopped being the nonaccomplishment mode.
• The endpoint, the signatures, and the tunnel ne'er beryllium successful the first place. Every relationship is consumer-initiated, and the loop runs down NAT, connected a laptop, aliases successful a scheduled job.

The provender could transportation 1 much thing. When a publication reaches the extremity of the log, the supplier could show you what should beryllium there: a count and a checksum of existent state, astatine the cursor you now hold. You comparison the two, and you know your replica is correct alternatively of assuming it. My 3 a.m. cron, the written confession, becomes a comparison I’ve already made by the clip I would person thought to schedule one.

The 4th time

If a provender for illustration that existed, the 4th clip I build this strategy would beryllium a loop: GET the feed, fto “upsert” upsert the entity into my db and “delete” delete it, and prevention the past cursor. That’s 20 lines pinch nary route, nary secrets to rotate, nary queue, and nary cron. The replica carries its ain impervious of correctness, and erstwhile personification asks which customers person an progressive subscription and a bouncing email address, the reply is simply a JOIN crossed section tables pinch nary silent asterisk attached.

Nobody serves this today. That’s the catch, and it’s besides the point.

I wanted to spot whether the thought survives being written down precisely, truthful I drafted it arsenic a protocol: SCROLL, short for Synchronized Change Replication Over Line Logs, astatine welidev.github.io/scroll. It’s draft-00 successful the request-for-comments consciousness of the phrase. It pins down the feed, the cursors, the streaming and polling modes, the checkpoints, tombstones, and retention, and it marks the places wherever my ain assurance is lowest. It besides doesn’t require waiting for providers, since a shim tin synthesize a provender from immoderate provider’s existing webhooks and database APIs, which is really I scheme to find retired wherever the creation is wrong.

If you’ve lived successful the valley, if you’ve written a dedup array aliases debugged a reconciliation cron aliases watched a delete evaporate, publication it and show maine wherever it breaks. Disagreement is the desired response; soundlessness is the nonaccomplishment mode.