Why Your Webhooks Are Failing: Debugging Timeouts & Asynchronous Processing

The most common reason webhooks fail is that your handler takes too long to respond, the provider retries the delivery, and you end up processing the same event multiple times — or not at all, once the retries run out. The fix is to split the handler into two parts: one that receives the event and returns immediately, and one that does the actual work asynchronously. Every other webhook reliability problem is secondary to this one.

I know this because I ignored it long enough to lose 8 payment events during a weekend promotion. Here’s what happened and how I fixed it.

---

How Providers Actually Handle Timeouts

Every webhook provider sets a window in which your server must return a successful response. If you don’t respond in time, they mark the delivery as failed and schedule a retry. The window varies by provider:

• Stripe: 30 seconds, but their documentation recommends responding within 5 seconds. Retries follow an exponential schedule over 72 hours — 15 attempts total.
• Shopify: 5 seconds. Retries every hour for 48 hours.
• GitHub: 10 seconds. One retry, 1 hour later.
• Twilio: 15 seconds. Up to 11,000 retries over 54 hours, which sounds generous until the same event fires your downstream logic thousands of times.
• SendGrid (inbound parse): 20 seconds. Three retries at 5-minute intervals.

Notice that the retry behavior differs wildly. Stripe is patient and will keep trying for three days. GitHub gives you one shot. For any mission-critical webhook — payment confirmation, subscription cancellation, user provisioning — you need to understand your provider’s retry schedule before you find out what it does during an outage.

The other thing providers don’t tell you clearly: a timeout is not just a slow response. It’s also a dropped connection, a 5xx response, or anything that isn’t a 2xx. Return a 500 because your database query failed and Stripe treats that as a failed delivery. Return nothing because your server crashed and the retry clock starts immediately.

---

What Was Actually Wrong in My Setup

The webhook in question was a Stripe payment_intent.succeeded event. When a payment cleared, the handler did the following in sequence:

1. Wrote the subscription record to Postgres
2. Called the HubSpot API to create or update a contact
3. Called SendGrid to send the welcome email
4. Made an outbound call to an n8n workflow webhook to trigger onboarding

Each step was fast in isolation. Postgres write: around 80ms. HubSpot API call: 340ms average. SendGrid: 210ms. The n8n workflow trigger: 290ms.

Add them up with a bit of overhead and the handler was averaging 1.1 seconds under normal conditions. Fine. Still well inside Stripe’s 30-second window.

Then we ran a promotion. New signups spiked. Stripe was delivering webhooks simultaneously. My database connection pool had a maximum of 10 connections. With 14 concurrent webhook deliveries — a mix of new payments and Stripe retrying earlier ones — every handler was waiting for a connection slot that wasn’t coming.

Average handler response time that night went from 1.1 seconds to 11.4 seconds, measured in Sentry. Some requests timed out entirely. Stripe logged them as failures and queued retries. The retries added more concurrent requests, which further saturated the pool. Over six hours, 34 webhook deliveries failed. Eight of them exhausted Stripe’s retry window before I noticed, which meant 8 successful payments that my system never processed.

The frustrating part: nothing was crashing. No 500 errors. The handler was running fine — it just couldn’t get a database connection fast enough. Stripe saw silence and assumed failure.

---

The Correct Pattern: Acknowledge First, Process Second

The handler’s only job should be:

1. Validate the webhook signature
2. Write the raw payload to a queue
3. Return 200

Everything else — the database write, the API calls, the downstream integrations — moves into a worker that processes jobs off the queue. The webhook delivery is confirmed in under 200ms. The actual work happens separately, at whatever pace your infrastructure supports.

This is not a complex pattern. Here’s a minimal FastAPI implementation:

python

import hmac
import hashlib
import json
import redis
from fastapi import FastAPI, Request, HTTPException

app = FastAPI()
queue = redis.Redis(host="localhost", port=6379, db=0)

STRIPE_WEBHOOK_SECRET = "whsec_your_secret_here"

def verify_stripe_signature(payload: bytes, sig_header: str, secret: str) -> bool:
    parts = dict(item.split("=", 1) for item in sig_header.split(","))
    timestamp = parts.get("t", "")
    signature = parts.get("v1", "")
    signed_payload = f"{timestamp}.{payload.decode('utf-8')}"
    expected = hmac.new(
        secret.encode("utf-8"),
        signed_payload.encode("utf-8"),
        hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature)

@app.post("/webhooks/stripe")
async def stripe_webhook(request: Request):
    payload = await request.body()
    sig_header = request.headers.get("stripe-signature", "")

    if not verify_stripe_signature(payload, sig_header, STRIPE_WEBHOOK_SECRET):
        raise HTTPException(status_code=400, detail="Invalid signature")

    # Push raw payload onto the queue — nothing else
    queue.lpush("webhook_jobs", payload)

    return {"status": "queued"}  # Returns 200 immediately

The worker that processes the queue:

python

import time
import json
import redis
import psycopg2

queue = redis.Redis(host="localhost", port=6379, db=0)

def process_payment_succeeded(event: dict):
    payment_intent = event["data"]["object"]
    event_id = event["id"]

    # Idempotency check — more on this below
    with get_db_connection() as conn:
        cursor = conn.cursor()
        cursor.execute(
            "SELECT 1 FROM processed_webhook_events WHERE event_id = %s",
            (event_id,)
        )
        if cursor.fetchone():
            print(f"Already processed {event_id}, skipping")
            return

        # Do the actual work
        cursor.execute(
            "INSERT INTO subscriptions (customer_id, status, created_at) VALUES (%s, %s, NOW())",
            (payment_intent["customer"], "active")
        )
        cursor.execute(
            "INSERT INTO processed_webhook_events (event_id, processed_at) VALUES (%s, NOW())",
            (event_id,)
        )
        conn.commit()

    # Downstream API calls happen here, outside the DB transaction
    create_hubspot_contact(payment_intent["customer"])
    send_welcome_email(payment_intent["customer"])
    trigger_onboarding_workflow(payment_intent["customer"])

def worker():
    print("Worker started")
    while True:
        _, raw = queue.brpop("webhook_jobs")
        event = json.loads(raw)

        if event.get("type") == "payment_intent.succeeded":
            process_payment_succeeded(event)

        time.sleep(0.1)

if __name__ == "__main__":
    worker()

After this change, webhook handler response time went to 180ms. The queue processed the full downstream chain — Postgres write, HubSpot, SendGrid, n8n — in an average of 4.3 seconds per event. Stripe never sees that 4.3 seconds. It sees a 180ms acknowledgment and moves on.

---

The Part That Will Still Bite You: Idempotency

Moving to async processing doesn’t eliminate the retry problem. It shifts it. Stripe will still retry if your acknowledgment gets lost in transit, if there’s a brief network blip, or if you deploy mid-delivery. Your worker will receive the same event more than once, and if you haven’t built idempotency in, you’ll send duplicate emails, create duplicate records, and charge duplicate fees.

The processed_webhook_events table in the code above is the idempotency layer. Before doing any work, the worker checks whether it has already processed this event ID. If it has, it skips the job silently. This is the correct behavior — not an error, not a warning, just a skip.

Two things worth being precise about here:

The event ID needs to come from the provider, not from something you generate. Stripe’s event IDs look like evt_1NxYzBLkdIwHu7ixAbcDef12. GitHub’s delivery IDs are UUIDs in the X-GitHub-Delivery header. Use whatever your provider sends.

The idempotency check and the main write need to happen in the same database transaction. If you check, then write, with a gap in between, two concurrent workers processing the same retry can both pass the check before either of them commits, and you’re back to duplicates. The check-and-insert should be one atomic operation.

---

The Connection Pool Problem (and How to Catch It)

The root cause of my specific incident wasn’t the handler logic — it was the connection pool. The fix for async processing helps here because workers can share a connection pool across jobs sequentially rather than 14 handlers all requesting connections simultaneously. But if you don’t adjust the pool size, you’ll hit the same ceiling under enough concurrency.

After moving to async, I kept the same pool size (10 connections) but changed how I managed them. The worker uses a single long-lived connection per worker process and reconnects on failure, rather than acquiring a new connection per job. For higher-volume scenarios, pgBouncer in transaction mode is the right answer — it multiplexes hundreds of application connections over a small number of actual Postgres connections.

The signal to watch for: handler latency that spikes exactly when concurrent webhook delivery increases, with no corresponding CPU or memory pressure. That’s connection pool saturation. It looks like a slowdown, not an error, which is why it’s easy to miss until the timeouts start.

---

---

Debugging Checklist When Webhooks Start Failing

When a webhook endpoint starts producing failures and you’re not sure why, this is the order I work through:

1. Check the provider’s delivery logs first, not your own logs. Stripe’s webhook dashboard shows exactly what response code you returned and how long the delivery took. If your logs show the handler ran successfully but Stripe shows a timeout, the discrepancy tells you the handler returned a 200 after Stripe had already given up waiting. You need to respond faster.

2. Look at the response time trend, not just failures. In Sentry or Datadog, pull the p95 response time for your webhook endpoint over the past 7 days. A gradual creep from 800ms to 4 seconds to 12 seconds tells a different story than a sudden spike. Gradual creep is usually database query degradation or a slow third-party API call in the critical path. A sudden spike is usually load or deployment-related.

3. Test your signature verification in isolation. Failed signature checks return 400, which providers log as a client error (not a server error). This is easy to accidentally trigger after rotating a webhook secret without updating your environment variable. Takes five minutes to verify and rules out a whole category of failures.

4. Replay a failed event to test the fix. Stripe lets you resend any webhook event from the dashboard. After deploying the async pattern, replay one of the previously-failed events and confirm it returns 200 within the expected window. Don’t assume the fix worked because you deployed it — actually verify.

5. Set up an alert on 4xx/5xx rates from your webhook endpoint. This should have been in place before the incident, but it usually isn’t. A simple alert that fires when the error rate on /webhooks/* exceeds 2% over 10 minutes will catch most problems before they exhaust retry windows.

---

What About n8n Webhook Triggers?

If you’re running webhooks into n8n workflows, the same problem applies with an added wrinkle: n8n’s webhook trigger nodes are synchronous by default. The provider waits for the entire workflow to complete before it gets a response. For a workflow that does database lookups, calls external APIs, and branches based on conditions, that can easily run 5–15 seconds — long enough to time out Shopify and GitHub reliably.

The solution in n8n is to use the webhook node’s “Respond to Webhook” node early in the workflow, before the heavy processing steps. This tells n8n to return the 200 immediately, then continue executing the rest of the workflow in the background. It’s the same acknowledge-first pattern, implemented at the workflow level rather than the code level.

I wrote about n8n workflow architecture separately, but this is the most operationally important thing to know if you’re routing production webhooks through n8n: never let the workflow completion time become the webhook response time.

---

FAQ

Why does my webhook work fine locally but time out in production? Two likely reasons. First, your local environment doesn’t have the latency that a real database and real third-party API calls introduce in production — a HubSpot API call that takes 40ms on localhost might take 380ms in production depending on network routing. Second, your local test usually sends one webhook at a time, while production might deliver several concurrently, which is when connection pool and resource contention problems appear.

Should I return 200 or 202 from a webhook handler? Semantically, 202 Accepted is more correct — it means “I received this and will process it, but haven’t finished yet.” In practice, most providers accept any 2xx as a successful delivery. Stripe, Shopify, and GitHub all treat 200 and 202 identically. Use 202 if you want to be precise; use 200 if you want to be safe across all providers.

How do I handle a webhook event that fails during async processing? This is why the queue exists. If the worker throws an exception while processing a job, the job should go back on the queue rather than being dropped. Redis-based queue libraries like RQ (Python) or BullMQ (Node.js) handle this automatically with a dead-letter queue for jobs that fail repeatedly. Check your dead-letter queue regularly — jobs there represent events that were received successfully but couldn’t be processed, and they need manual investigation.

My provider doesn’t send an event ID. How do I implement idempotency? Derive a deterministic ID from the event content. For a payment confirmation, you might hash the customer ID + amount + timestamp to create a stable fingerprint. The fingerprint won’t be unique if the same customer makes two identical payments within the same second, but in practice that edge case is rare enough to handle manually if it occurs. The alternative — storing the full payload and doing a content-based deduplication check — is more robust but slower.

How many workers should I run? Start with two and measure queue depth over time. If jobs are accumulating faster than they’re being processed, add workers. If the queue stays near zero, one worker is probably sufficient and two gives you redundancy. The limit is usually your database connection pool size divided by connections per worker, not CPU. A worker that holds one connection and processes jobs sequentially is more efficient than four workers each holding their own connection and competing for the same pool slots.

What’s the difference between a webhook timeout and a webhook signature failure? A timeout means your server didn’t respond within the provider’s window. A signature failure means your server responded with a 4xx because the HMAC signature on the incoming request didn’t match what you expected. Timeouts show up as provider-side errors in the delivery logs; signature failures show up as your server explicitly rejecting the request. Both result in retries from most providers, but signature failures are usually a configuration problem (wrong secret, secret rotation not propagated) rather than a load problem.

---

The async pattern is the right default for any webhook endpoint that does more than log the event. The overhead of adding a queue is an hour of setup; the cost of not having one shows up as lost events at the worst possible time — high traffic, a promotion running, users expecting immediate confirmation of a payment they just made. I learned this the uncomfortable way. The eight events that exhausted Stripe’s retry window that night took about three days to reconstruct manually from Stripe’s API and reconcile with the database. The async setup took two hours to build.