The Infinite OAuth Loop: Automating Token Refreshes in Custom Webhooks

The infinite OAuth loop is what happens when your webhook handler detects an expired access token, fires a refresh request, stores the new token, and then a second concurrent handler — running simultaneously against the same integration — does the exact same thing half a second later. If your OAuth provider rotates refresh tokens on each use, that second refresh just invalidated the token the first handler stored. Now neither handler has a valid token. Both will try to refresh again. The loop is now live.

I ran 7 webhook integrations that required OAuth token management for about four months without addressing this properly. In that period I had 23 token expiry incidents — situations where a webhook integration silently stopped working because token state had become invalid. Most of them happened between midnight and 6am when background jobs created concurrent load that my daytime testing never replicated. After building a proper distributed locking solution around token refresh, the incident count dropped to zero and has stayed there for five months.

This post covers the race condition in detail, the Redis-based distributed lock that fixes it, and a proactive refresh strategy that prevents the problem from occurring in the first place. There’s also a provider comparison table because the way different OAuth implementations handle refresh token rotation is one of the more chaotic parts of this problem space.

How Token Refresh Is Supposed to Work

OAuth 2.0 access tokens are short-lived by design. Most providers issue them with a 1-hour expiry. When a webhook handler needs to call an API and the access token is expired, the standard refresh flow is straightforward:

import requests
import time
import json

def get_valid_token(token_store: dict) -> str:
    """Returns a valid access token, refreshing if necessary."""

    # Check if current token is still valid (with 60s buffer)
    if token_store["expires_at"] > time.time() + 60:
        return token_store["access_token"]

    # Token is expired or expiring soon — refresh it
    response = requests.post(
        token_store["token_endpoint"],
        data={
            "grant_type": "refresh_token",
            "refresh_token": token_store["refresh_token"],
            "client_id": token_store["client_id"],
            "client_secret": token_store["client_secret"],
        }
    )
    response.raise_for_status()
    new_tokens = response.json()

    # Update the store
    token_store["access_token"] = new_tokens["access_token"]
    token_store["expires_at"] = time.time() + new_tokens["expires_in"]

    # CRITICAL: store new refresh token if provider rotates them
    if "refresh_token" in new_tokens:
        token_store["refresh_token"] = new_tokens["refresh_token"]

    return token_store["access_token"]

# This works perfectly — for a single process.
# The moment you run two webhook workers simultaneously, it breaks.

In a single-process environment, that flow works fine. The problem is that production webhook infrastructure almost never runs as a single process. You run multiple workers for concurrency. Workers process events in parallel. And when a batch of webhook events arrives simultaneously — which happens constantly in any real integration — multiple workers independently check the token, independently detect it’s expired, and independently attempt to refresh it.

The Race Condition That Creates the Loop

I was running 4 concurrent webhook worker processes. Average token refresh completes in about 340ms. The critical window — between one worker detecting token expiry and completing the refresh — is long enough for other workers to independently detect the same expired token and queue their own refresh attempts. With 4 workers all processing events from the same integration simultaneously, all 4 will often detect the expired token within milliseconds of each other.

With providers that issue a new refresh token on every refresh (token rotation), the sequence that causes the loop looks like this:

T=0ms    Worker 1: reads token from DB → expired
T=1ms    Worker 2: reads token from DB → expired
T=2ms    Worker 3: reads token from DB → expired
T=2ms    Worker 4: reads token from DB → expired

T=3ms    Worker 1: POST /token with refresh_token=ABC → pending
T=3ms    Worker 2: POST /token with refresh_token=ABC → pending
T=3ms    Worker 3: POST /token with refresh_token=ABC → pending
T=3ms    Worker 4: POST /token with refresh_token=ABC → pending

T=340ms  Worker 1: receives access_token=NEW1, refresh_token=XYZ
         Provider has now invalidated ABC and issued XYZ as current refresh token.

T=341ms  Worker 2: receives 400 invalid_grant (ABC was already rotated)
T=341ms  Worker 3: receives 400 invalid_grant
T=341ms  Worker 4: receives 400 invalid_grant

# Workers 2, 3, and 4 now have no valid token.
# They detect failure, read from DB (Worker 1 may not have written yet),
# find an expired token, and attempt to refresh again.
# Worker 1 writes XYZ to DB.
# Workers 2-4 read XYZ from DB, POST /token with refresh_token=XYZ.
# Provider rotates XYZ → issues ZZZ, invalidates XYZ.
# One worker gets ZZZ. The others get invalid_grant again.
# Loop continues until the background task gives up or triggers an alert.

In my setup the loop self-terminated after 3–5 iterations because my retry logic had a maximum attempt count. What was left was a completely invalid token state: the refresh token in the database was whatever the last successful rotation had issued, but none of the workers had successfully completed their original webhook action. The integration was broken until someone manually re-authorized or the next scheduled token check ran and happened to be the only process running at that moment.

The Part Nobody Documents Properly: Provider Inconsistency

Before getting to the fix, I want to name the thing that made this significantly harder to debug and solve than it should have been: OAuth providers implement token rotation differently, document it inconsistently, and change their behavior with configuration options that aren’t always surfaced where you’d look for them.

Google is the worst example of this. Their refresh tokens do not rotate on use — which sounds like it makes concurrent refresh safe, and it mostly does. But Google refresh tokens expire after 7 days if your OAuth application is in “testing” status in Google Cloud Console. Not “in development mode,” not “unverified” — specifically the “Testing” publishing status under OAuth consent screen settings. This is documented in a footnote in one of their OAuth guides. It is not mentioned in the invalid_grant error documentation. It is not surfaced in the error response. The first time this happened to me, one of my integrations stopped working exactly 7 days after the initial authorization and I spent an hour and a half looking at my refresh logic before someone mentioned the testing status issue in a Stack Overflow comment from 2021.

This is not a minor documentation gap. It’s a behavior that causes production integrations to fail on a 7-day cadence for anyone who hasn’t moved their app to production status, and the failure mode is completely silent until a user reports that the integration stopped working. Google should surface this in the error response. They don’t.

The Fix: Distributed Lock Around Token Refresh

The solution to the race condition is to ensure only one process can execute the token refresh at a time. Every other process waits, reads the freshly stored token, and uses that. A Redis distributed lock using SET NX EX handles this cleanly.

import redis
import time
import uuid
import requests

r = redis.Redis(host="localhost", port=6379, db=0)

LOCK_TTL_SECONDS = 30      # Max time a refresh can hold the lock
LOCK_WAIT_INTERVAL = 0.05  # 50ms polling interval while waiting
LOCK_MAX_WAIT = 10         # Give up waiting after 10 seconds

def acquire_refresh_lock(integration_id: str) -> str | None:
    """
    Attempts to acquire the token refresh lock for this integration.
    Returns a unique lock token if acquired, None if lock is already held.
    """
    lock_key = f"token_refresh_lock:{integration_id}"
    lock_value = str(uuid.uuid4())  # Unique per acquisition

    # SET key value NX EX seconds
    # NX = only set if key doesn't exist
    # EX = expire after LOCK_TTL_SECONDS (prevents deadlock if process crashes)
    acquired = r.set(lock_key, lock_value, nx=True, ex=LOCK_TTL_SECONDS)
    return lock_value if acquired else None

def release_refresh_lock(integration_id: str, lock_value: str):
    """Release the lock — only if we still own it (prevents releasing another process's lock)."""
    lock_key = f"token_refresh_lock:{integration_id}"
    # Lua script ensures atomic check-and-delete
    lua_script = """
    if redis.call("get", KEYS[1]) == ARGV[1] then
        return redis.call("del", KEYS[1])
    else
        return 0
    end
    """
    r.eval(lua_script, 1, lock_key, lock_value)

def get_valid_token(integration_id: str, db) -> str:
    """
    Get a valid access token, using distributed locking to prevent
    concurrent refresh race conditions.
    """
    token_data = db.get_token(integration_id)

    # Token is still valid
    if token_data["expires_at"] > time.time() + 60:
        return token_data["access_token"]

    # Token needs refresh — try to acquire the lock
    lock_value = acquire_refresh_lock(integration_id)

    if lock_value:
        # We hold the lock — perform the refresh
        try:
            new_tokens = _perform_refresh(token_data)
            db.update_token(integration_id, new_tokens)
            return new_tokens["access_token"]
        finally:
            release_refresh_lock(integration_id, lock_value)
    else:
        # Another process holds the lock — wait for it to finish
        deadline = time.time() + LOCK_MAX_WAIT
        while time.time()  time.time() + 60:
                return token_data["access_token"]  # Fresh token is available

        raise TimeoutError(
            f"Waited {LOCK_MAX_WAIT}s for token refresh on {integration_id}. "
            "The refreshing process may have crashed."
        )

def _perform_refresh(token_data: dict) -> dict:
    response = requests.post(
        token_data["token_endpoint"],
        data={
            "grant_type": "refresh_token",
            "refresh_token": token_data["refresh_token"],
            "client_id": token_data["client_id"],
            "client_secret": token_data["client_secret"],
        },
        timeout=15
    )
    response.raise_for_status()
    result = response.json()

    return {
        "access_token": result["access_token"],
        "refresh_token": result.get("refresh_token", token_data["refresh_token"]),
        "expires_at": time.time() + result["expires_in"],
    }

Two details in this implementation that matter more than they look:

The lock TTL of 30 seconds isn’t arbitrary. It needs to be long enough that a legitimate slow refresh (provider API under load, network variance) doesn’t cause the lock to expire and release before the refresh completes — which would allow a second process to start a concurrent refresh we were trying to prevent. It needs to be short enough that if the process holding the lock crashes, the lock self-releases and recovery is fast. 30 seconds is conservative given that a real refresh rarely takes more than 2 seconds, but I’d rather have a 28-second wait in a crash scenario than a failed refresh because a stressed provider API took 25 seconds.

The Lua script for lock release is critical. A naive implementation would read the lock value, check if it matches, then delete. Between the read and the delete, another process could acquire the lock (if the TTL expired), and your delete would release their lock, not yours. The Lua script executes atomically on the Redis server, making the check-and-delete a single indivisible operation.

Proactive Refresh: Prevent the Problem Before It Starts

The distributed lock solves the race condition when it occurs. Proactive refresh prevents it from occurring at all in most cases. Instead of waiting for a token to expire and then refreshing reactively, you refresh on a schedule — before the expiry window, when concurrency is low and predictable.

import schedule
import time

REFRESH_BUFFER_MINUTES = 15  # Refresh when token has 15 minutes remaining

def refresh_all_expiring_tokens(db, redis_client):
    """
    Runs on a schedule. Finds all tokens expiring within the buffer window
    and refreshes them before webhook workers encounter expiry.
    """
    expiry_threshold = time.time() + (REFRESH_BUFFER_MINUTES * 60)
    expiring_integrations = db.get_tokens_expiring_before(expiry_threshold)

    for integration in expiring_integrations:
        # Still use the lock — in case a webhook worker is refreshing concurrently
        lock_value = acquire_refresh_lock(integration["id"])
        if not lock_value:
            continue  # Another process already refreshing this one

        try:
            new_tokens = _perform_refresh(integration)
            db.update_token(integration["id"], new_tokens)
            print(f"Proactively refreshed token for {integration['id']}")
        except Exception as e:
            print(f"Proactive refresh failed for {integration['id']}: {e}")
            # Don't crash the whole job — log and move on
            # Alert if this integration has failed N consecutive refreshes
        finally:
            release_refresh_lock(integration["id"], lock_value)

# Run every 5 minutes — catches tokens expiring within the 15-minute buffer
schedule.every(5).minutes.do(refresh_all_expiring_tokens, db=db, redis_client=r)

while True:
    schedule.run_pending()
    time.sleep(10)

With a 15-minute buffer and a 5-minute check interval, tokens are refreshed while they’re still valid. Webhook workers arrive to find a fresh token in storage and skip the refresh entirely. The race condition can still theoretically occur — if the scheduler runs at the same time as a high-concurrency webhook burst — but in practice the scheduler runs in a single process and the lock prevents any conflicts.

The Results From My Own Setup

The monitoring I built around this tracks two metrics: token refresh attempts per integration per day, and token refresh failures. Before implementing the distributed lock and proactive scheduler, my 7 integrations generated a combined 23 token expiry incidents over 4 months. After the implementation, that number has been zero for 5 months.

The more specific data point: in the month before I fixed this, my HubSpot integration was generating an average of 1.4 failed refresh attempts per day — almost all between 11pm and 4am when the background sync jobs ran and generated concurrent load. The dashboard below shows that drop-off directly.

Provider Comparison: Token Rotation Behavior

This table is what I wish had existed before I started. The race condition severity varies significantly by provider depending on whether they rotate refresh tokens. Providers that don’t rotate are inherently more forgiving of concurrent refreshes.

The providers with high race condition risk (HubSpot, Zoom, Xero) are the ones where the distributed lock is non-optional. Google and Shopify are forgiving of naive implementations — which is probably why most OAuth tutorial code is written assuming Google-style behavior and then fails when applied to HubSpot.

Handling the Refresh Token Expiring Entirely

The distributed lock and proactive refresh handle the race condition and keep tokens fresh during normal operation. They don’t handle the case where the refresh token itself has expired — typically because the integration went unused long enough for the refresh token’s expiry to lapse (Xero’s 60-day window, or Google’s 7-day testing mode window).

When _perform_refresh() returns a 400 with invalid_grant and you’ve confirmed the issue isn’t the race condition (because you now have the lock), the refresh token is genuinely expired and you need to re-authorize from scratch. The handling for this case:

def _perform_refresh(token_data: dict) -> dict:
    response = requests.post(
        token_data["token_endpoint"],
        data={
            "grant_type": "refresh_token",
            "refresh_token": token_data["refresh_token"],
            "client_id": token_data["client_id"],
            "client_secret": token_data["client_secret"],
        },
        timeout=15
    )

    if response.status_code == 400:
        error = response.json().get("error", "")
        if error == "invalid_grant":
            # Refresh token is invalid or expired — cannot recover automatically.
            # Flag this integration as requiring re-authorization.
            raise RefreshTokenExpiredError(
                f"Refresh token invalid for integration {token_data['id']}. "
                "Manual re-authorization required."
            )

    response.raise_for_status()
    result = response.json()

    return {
        "access_token": result["access_token"],
        "refresh_token": result.get("refresh_token", token_data["refresh_token"]),
        "expires_at": time.time() + result["expires_in"],
    }

class RefreshTokenExpiredError(Exception):
    """Raised when a refresh token is invalid and manual re-auth is needed."""
    pass

# In your worker or proactive refresh job:
try:
    token = get_valid_token(integration_id, db)
except RefreshTokenExpiredError:
    db.set_integration_status(integration_id, "requires_reauth")
    notify_ops_team(
        f"Integration {integration_id} requires re-authorization. "
        "Refresh token expired or revoked."
    )
    # Don't retry — re-auth requires a human action
    return

The requires_reauth status flag is useful for two things: it prevents the proactive refresh job from repeatedly attempting a refresh it knows will fail, and it gives you a clean signal to surface in any integration health dashboard you’re running. An integration in requires_reauth status should trigger an alert to whoever owns the integration, not just a log entry.

FAQ

Why use Redis for the lock instead of a database row-level lock?

Database advisory locks work but add latency, require careful transaction handling, and don’t automatically expire if the process holding them crashes without releasing cleanly. Redis SET NX EX is atomic (the check-and-set is a single operation, unlike database SELECT-then-INSERT patterns), sub-millisecond, and automatically self-releases after the TTL even if the holding process dies. If you don’t have Redis in your stack, a database-based approach works — use INSERT INTO refresh_locks (...) ON CONFLICT DO NOTHING in Postgres for atomic lock acquisition, and a separate cleanup job to release locks held longer than your TTL.

What if the proactive refresh job itself crashes mid-refresh?

The lock TTL handles this. If the job crashes after acquiring the lock but before releasing it, the lock expires after 30 seconds and the next run of the scheduler can acquire it and retry. The database token state at that point depends on where in _perform_refresh() the crash occurred: if it crashed before the DB write, the old token is still in place; if it crashed after, the new token was written correctly. The worst case is a crash between receiving the new tokens from the provider and writing them to the database — which leaves you with an invalidated refresh token in storage. This is worth monitoring with consecutive failure alerts: if an integration fails refresh 3 times in a row, escalate immediately regardless of the error type.

Should I store tokens in Redis or in the database?

Database, with Redis used only for the lock. Tokens are the durable credential that allows access to a user’s data. Storing them only in Redis risks data loss on Redis restart or eviction (if Redis is configured with an eviction policy for memory management). The database gives you durability, backup coverage, and a clean audit trail of when tokens were last refreshed. Redis is fast but ephemeral — using it as a lock store is appropriate; using it as your only token store is a reliability risk most teams don’t intend to take.

How do I handle a provider that requires PKCE for token refresh?

PKCE (Proof Key for Code Exchange) applies to the initial authorization flow, not the token refresh. The refresh flow for PKCE-enabled apps uses the same grant_type: refresh_token POST — the code_verifier used during initial authorization is not required for subsequent refreshes. Some providers (particularly those following newer OAuth 2.1 drafts) require the client_id to be included in the refresh request even for confidential clients; check your specific provider’s refresh token documentation rather than assuming the standard parameters are sufficient.

Is there a way to know which provider is causing the most problems before implementing this?

Log the provider name alongside every token refresh attempt and failure before you start. Even a week of that data tells you which integration accounts for the majority of incidents, which lets you prioritize the lock implementation for high-rotation providers (HubSpot, Zoom, Xero) over low-rotation ones (Google, Shopify). The pattern in my setup was clear: 19 of my 23 incidents were from two integrations, both with providers that rotate refresh tokens on every use. The other 5 integrations were effectively never a problem and could have waited for a second implementation pass.

Can n8n or Zapier handle OAuth token refresh automatically so I don’t need to build this?

For integrations that live entirely within n8n or Zapier, yes — both platforms manage OAuth credentials and handle token refresh internally. The problem this post describes applies when you’re running custom webhook handlers outside those platforms, using OAuth credentials directly in your own code. If you’re building a webhook processor in Python or Node.js that calls an OAuth-protected API, you own the token lifecycle. If you’re routing webhooks through n8n and using n8n’s built-in credential system to make downstream API calls, n8n handles the refresh for you — though n8n has its own token refresh timing behavior worth understanding if you’re running high-concurrency workflows.