The Silent Stripe Webhook Signature Bug in AI Billing Systems
Your webhook signature check can pass every test you write for it and still be wrong in production.
That's not a hypothetical. It happened in a live dual-rail billing system — Stripe and PayPal running side by side — found and fixed the same day it surfaced, no client involved, no invented numbers, just a bug class worth knowing about.
The old approach to webhook security was: verify the signature, log success, move on. The modern approach is: verify against the exact bytes Stripe or PayPal sent — not a copy your own framework re-serialized first.
If you haven't read the operational side of this yet, start here: AI Governance for Small Teams: A Practical Policy. This post is the engineering companion — the kind of hole that never shows up in a policy document, only in a payment log nobody is watching.
The Bug That Passes Its Own Test Suite
Most webhook-security writing stops at "verify the signature." That's a checkbox, not a control. The bug class that actually costs money is quieter: a signature check that only ever gets exercised against data your own framework has already touched.
- Stripe or PayPal sends a request with a raw body and a signature header computed over those exact bytes.
- Somewhere upstream — body-parsing middleware, a framework default, a logging wrapper — the body gets parsed into an object and turned back into a string before your verification code sees it.
- Parsing and re-encoding JSON does not reliably reproduce the original bytes: key order, whitespace and number formatting can all shift.
- Your check runs against the re-encoded string; the signature was computed over the original. They don't match on real production traffic — and Stripe or PayPal log the delivery as sent successfully, because from their side, it was.
Why the Test Suite Doesn't Catch It
Unit tests usually build the request body as an object, serialize it once, sign it, and send it straight through — no upstream middleware to introduce a second serialization pass. The test proves your logic is implemented correctly; it never sees the exact bytes a real request delivers. Green build, broken production path. It's the same failure shape our RAG explainer describes for AI answers that look right until they silently aren't.
What This Actually Costs You (Silently)
- No error appears in your logs. The check fails, you return a 400, Stripe or PayPal retry and give up — logged as delivered-but-rejected on their side, not yours.
- A customer pays and doesn't get upgraded. The charge succeeded; the webhook that flips their plan never processed.
- It looks intermittent, not systemic. The bug depends on whichever middleware touches the body first, so it can hit one route and not another.
Where Stripe and PayPal Diverge (and Why That Doubles the Risk)
Stripe signs the raw body with an HMAC you verify locally — see Stripe's signature-verification documentation. PayPal instead has you call its webhook verification API with the headers and body you received. Two verification models means two places the same mistake can hide.
Step 1: Verify Against Raw Bytes, Not Parsed Objects
Capture the body as a raw buffer or string before any JSON-parsing middleware runs, and verify against that capture — never against the result of parsing and re-encoding it. Most frameworks let you opt a route out of automatic body parsing; use that for every webhook endpoint.
Step 2: Make the Retry Safe With an Idempotency Key
Once verification is correct, the next failure mode is a webhook delivered twice that applies its effect twice. Use the event ID as an idempotency key, store which IDs you've already processed, and make the handler a no-op on a repeat. Related reading: Building Resilient Digital Products covers this pattern beyond billing.
Step 3: Log the Raw Payload Before You Touch It
Persist the raw body and headers for every webhook, verified or not, before any processing. When a discrepancy surfaces days later, a raw log is the only way to reconstruct what actually arrived. Treat it like the audit trail for any system making automated decisions: see AI Second Brains at Work.
Step 4: Alert on Verification Failures — Don't Just Retry
A signature-verification failure should never be a silent 400. Route it to an alert, not just a log line, and separate "verification failed" from "handler threw an error" — different problems, different responses. If AI-driven support triages your tickets, a billing desync should escalate to a human fast; see Customer Support AI That Actually Works.
An idempotent, correctly verified webhook handler produces the same, correct result no matter how many times a delivery is retried, replayed, or arrives out of order — and it fails loudly, not silently, when it can't verify what it received.
A Four-Week Rollout Plan
Week 1 — Audit every endpoint for where raw-body capture happens, then reproduce the bug by round-tripping a captured payload through parse-then-stringify.
Week 2 — Fix raw-byte verification on the highest-value webhook first, usually the one gating plan upgrades, and add idempotency-key storage with a no-op path for repeats.
Week 3 — Add structured logging of every raw payload, verified or rejected, and wire verification failures to an alert channel.
Week 4 — Run a replay test against each endpoint and confirm the no-op path holds. Document the verification model per provider — worth treating as a system; see this problem-solving framework.
What to Measure
- Webhook verification failure rate, tracked separately from general application errors.
- Time between a payment event and the corresponding plan change, with an alert threshold.
- Duplicate-delivery rate your idempotency layer absorbs — nonzero means retries are already happening whether you're watching or not.
- Count of webhooks with no raw-payload log, which should be zero.
If You Want ZPro to Build This With You
At Zdravevski Professionals, the AI Systems Audit exists for exactly this kind of hole — the one that doesn't show up until something fails. We look at:
- how your billing and payment webhooks actually verify, not how the integration guide assumed they would
- where retries, duplicates and idempotency are handled versus assumed
- what gets logged before a payload is touched, so an incident is reconstructable
- where a silent failure should have been a loud one