If an incident review at your company ever ended with “the database blipped for twenty seconds, so why were we down for forty minutes?”, this post is for you. Retry storms are usually the answer. A dependency slows down, every client retries, the retries add load, the dependency slows down further, and the system stays broken long after the original trigger is gone. This guide walks through the failure modes in the order they tend to compound, from a retry loop with no backoff to a recovery that never arrives. For each one you get the library default that causes it, the configuration that fixes it, and the signal that tells you it is happening. By the end, you should be able to read your own retry policies, layer by layer, and predict how many requests a single user click becomes when something downstream fails.

Library defaults in this post were checked on 2026-10-20 against the AWS SDK retry behavior reference, the gRPC A6 retry design, the Envoy v3 API reference, the Istio 1.31 VirtualService reference, the Polly v8 docs, tenacity 9.2.1 and axios-retry 4.5.0. The code examples were run on the environment below.

Run 2026-10-20
Windows 11 Pro 10.0.26200, AMD Ryzen 5 8600G
Node.js 24.22.0, Python 3.13.16, tenacity 9.2.1, httpx 0.28.1

What Is a Retry Storm?

A retry storm is an overload caused by clients retrying failed requests faster than a struggling service can recover. Each failure produces more requests instead of fewer, so the extra load keeps the service failing. The storm sustains itself: it can continue after the original trigger, such as a slow query or a failover, has already ended.

That last property is what makes retry storms so expensive. Most outages end when the cause is fixed. In contrast, a retry storm can outlive its cause by an order of magnitude, because the cause is no longer the problem. The retries are.

Why Retry Storms Outlast the Trigger

Researchers call this class of incident a metastable failure. The term comes from Bronson et al. at HotOS 2021, and a follow-up paper, Metastable Failures in the Wild at OSDI 2022, studied 22 of them across 11 organizations. Its abstract states that at least 4 out of 15 major outages in the last decade at Amazon Web Services were caused by metastable failures. Retries are the textbook sustaining effect in that model.

The mechanism is a feedback loop. Suppose a service can handle 1,000 requests per second, and a brief stall pushes latency past client timeouts. Every timed-out request comes back as a retry, so offered load climbs while capacity is still reduced. Consequently, more requests time out, which produces more retries. Once the loop is running, removing the trigger does not help, because the service is now overloaded by its own clients.

In practice, this is why “restart the database” or “roll back the deploy” often fails to end the incident. The only exits are reducing the load, by shedding requests or turning retries off, or adding capacity faster than the retries grow. Both are much harder at minute thirty than they would have been with a sane retry policy at minute zero.

Retry Storms in Order: Quick Reference

Here is the whole sequence in one table. Each stage makes the next one worse, and most production incidents involve at least three of them at once.

Stage	Failure mode	Typical cause	Fix
1	Retries with no delay	Library default of zero wait	Exponential backoff with a cap
2	Synchronized retries	Backoff without jitter	Full jitter
3	Retries at every layer	Each hop has its own policy	Retry at one layer only
4	Timeouts shorter than the work	Per-attempt timeout restarts the clock	One deadline for the whole call
5	Retrying errors that mean stop	Retrying 429, 503 and non-idempotent writes	Honor Retry-After, retry only safe calls
6	Recovery that never arrives	No limit on retry volume	Retry budgets and load shedding

The rest of the post explains each row, including why it happens and how to stop it from happening again.

Stage 1: Retries With No Delay

The first failure mode is the simplest. A request fails, and the client sends it again immediately. Under normal conditions, nobody notices, because the second attempt usually succeeds. During an incident, however, the client turns into a load generator pointed at the service that is already struggling.

Surprisingly, several popular libraries ship with exactly this behavior. The tenacity documentation says it plainly: “the default behavior is to retry forever without waiting when an exception is raised.” Similarly, axios-retry defaults to retries: 3 with a retryDelay of noDelay, which returns zero. Neither default is a bug. They are reasonable for a unit test and dangerous in production.

What a Bare Retry Decorator Does to a Busy Service

This script points a bare @retry at a mock inventory service that returns 503 for its first 50 requests. Nothing else is configured:

# naive_retry_demo.py
import httpx
from tenacity import retry

calls = 0


def busy_for_50_requests(request: httpx.Request) -> httpx.Response:
    global calls
    calls += 1
    return httpx.Response(503 if calls <= 50 else 200)


client = httpx.Client(
    base_url="http://inventory",
    transport=httpx.MockTransport(busy_for_50_requests),
)


@retry  # library defaults: no stop condition, no wait
def get_inventory() -> httpx.Response:
    response = client.get("/inventory/A1")
    response.raise_for_status()
    return response


get_inventory()
stats = get_inventory.statistics
print(f"requests sent: {calls}")
print(f"attempt_number: {stats['attempt_number']}")
print(f"seconds spent waiting: {stats['idle_for']}")

python naive_retry_demo.py

Output, identical on each of three runs:

requests sent: 51
attempt_number: 51
seconds spent waiting: 0.0

One logical call became 51 requests with zero seconds of backoff between them. In this demo, the service recovered on request 51. In a real incident, nothing guarantees that, and the decorator would keep going for as long as the outage lasted.

Bounded Retries With Backoff

The fix is to give every retry policy three limits: which errors to retry, how many attempts to make, and how long to wait between them. Here is the same call with all three set:

import logging

import httpx
from tenacity import (
    before_sleep_log,
    retry,
    retry_if_exception,
    stop_after_attempt,
    stop_after_delay,
    wait_random_exponential,
)

logger = logging.getLogger(__name__)
RETRYABLE_STATUS = {502, 503, 504}


def is_transient(exc: BaseException) -> bool:
    # Connection errors, resets and timeouts are worth one more try
    if isinstance(exc, httpx.TransportError):
        return True
    # 4xx means the request is wrong; repeating it changes nothing
    return isinstance(exc, httpx.HTTPStatusError) and exc.response.status_code in RETRYABLE_STATUS


@retry(
    retry=retry_if_exception(is_transient),
    stop=stop_after_attempt(3) | stop_after_delay(10),
    wait=wait_random_exponential(multiplier=0.1, max=5),
    before_sleep=before_sleep_log(logger, logging.WARNING),
    reraise=True,
)
def get_inventory(client: httpx.Client, sku: str) -> dict:
    response = client.get(f"/inventory/{sku}", timeout=2.0)
    response.raise_for_status()
    return response.json()

Three attempts is the number most production defaults converge on. For instance, the AWS SDKs default to max_attempts of 3, and the Google SRE book describes a per-request retry limit of three attempts. Additionally, before_sleep_log matters more than it looks. Retries that leave no trace in the logs are the reason teams discover their retry policy during the postmortem instead of during the incident.

Stage 2: Synchronized Retries Without Jitter

Adding exponential backoff fixes the tight loop. However, it creates a subtler problem when many clients fail at the same moment. If a thousand clients all see a 503 at the same instant and all wait exactly 100 ms, the service receives a thousand retries at the same instant 100 ms later. Then it receives them again at 300 ms, and at 700 ms. The backoff spreads retries out over time for each client, but it keeps every client in lockstep with every other client.

This matters most after a shared event: a failover, a deploy, a network partition healing. Those are exactly the moments when many clients fail together. As a result, deterministic backoff turns one recovery into a series of synchronized waves, and each wave can knock the service back over.

Why Full Jitter Fixes It

Jitter randomizes each delay so that clients spread out across the backoff window. The AWS SDKs now document the formula they use as full jitter:

delay = random(0, 1) × min(cap, base_delay × 2^retry)

Each retry waits a uniformly random time between zero and the exponential ceiling. Because of this, a thousand clients produce a steady trickle of retries across the window instead of a spike at its edge. Marc Brooker’s Builders’ Library article on timeouts, retries and backoff explains the reasoning and is worth reading in full.

Notably, jitter is still off by default in some libraries. The Polly v8 retry strategy defaults to BackoffType of Constant, a 2-second Delay, and UseJitter set to False. In other words, a .NET service that adds Polly with default options gets fixed two-second retries in perfect lockstep across every instance. The wait_random_exponential call in the tenacity example above is full jitter, which is why it was chosen over wait_exponential.

Stage 3: Retries at Every Layer

This is the stage that turns a bad afternoon into a multi-hour outage. Real requests cross several services, and each service typically has its own retry policy. Retries at different layers do not add up. They multiply.

If each of three layers makes up to three attempts, a single failing call at the bottom receives up to 3 × 3 × 3 = 27 requests. The general formula is attempts raised to the number of retrying layers. The Builders’ Library article gives a five-layer example and warns that if each layer retries independently, “the load on the database will increase 243x, making it unlikely to ever recover.”

Attempts per layer	1 layer	2 layers	3 layers	4 layers
2	2	4	8	16
3	3	9	27	81
4	4	16	64	256

These are worst-case multipliers for a call that fails every time, which is exactly what happens when the bottom dependency is down.

Hidden Layers You Did Not Configure

Most teams can name the retry policy in their own code. Far fewer can name the ones they inherited. For example, the Istio VirtualService reference documents a default cluster-wide retry policy of attempts: 2 with retryOn: connect-failure,refused-stream,unavailable,cancelled, and the maximum number of requests is 1 + attempts. Therefore, every hop through an Istio mesh can already be a three-attempt layer for connection failures, even though nobody on the team wrote a retry.

The same applies to SDKs. An AWS SDK client makes three attempts by default, and DynamoDB clients make four. Wrapping an SDK call in your own retry decorator stacks a second layer on top of the first, and the multiplication starts in your own process.

The fix the SRE book and the Builders’ Library both recommend is to retry at a single point in the stack. In practice, that usually means the layer closest to the failing dependency retries, and every layer above it fails fast. The SRE book adds an “overloaded; don’t retry” error that lower layers return so that callers know the retry already happened. In a mesh, you can turn the hidden layer off for services that handle their own retries:

# Disable mesh-level retries where the application already retries
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: inventory
spec:
  hosts:
    - inventory
  http:
    - route:
        - destination:
            host: inventory
      retries:
        attempts: 0

Per the Istio reference, setting attempts to 0 disables retries. Keep mesh retries for connection-level failures on services without their own policy, since those are cheap and usually safe. However, pick one owner per hop and write it down.

Stage 4: Timeouts Shorter Than the Work

Timeouts and retries interact in a way that rarely shows up in code review. When a client times out, the server does not know. It keeps working on the abandoned request, holding a connection, a thread and perhaps a database transaction. Meanwhile, the client sends a fresh copy, and now the server does the same work twice.

If the client timeout is shorter than the server’s normal latency under load, every request becomes two or three requests’ worth of work, and none of them complete. This is the cruelest version of a retry storm, because the server is fully busy doing work nobody is waiting for. Consequently, its throughput of useful responses drops toward zero while its CPU sits at 100%.

One Deadline for the Whole Call

The fix has two parts. First, give the whole operation a single deadline and make every attempt share it, so that retries cannot extend a request past the point where the caller has given up. Second, propagate that deadline downstream, so the server can stop working on requests whose caller is gone. gRPC does this natively with deadlines. For HTTP, a header carrying the remaining time budget does the same job.

This TypeScript helper implements the deadline, full jitter, Retry-After handling and a retry budget in one place. It runs on Node.js 24 with no dependencies:

// retry.ts: bounded retries with full jitter, a deadline, and a retry budget
const RETRYABLE_STATUS = new Set([429, 502, 503, 504]);
const IDEMPOTENT_METHODS = new Set(["GET", "HEAD", "OPTIONS", "PUT", "DELETE"]);

// Token bucket in the style of gRPC retry throttling: failures cost 1 token,
// successes refund `ratio`, and retries stop below half of `maxTokens`.
export class RetryBudget {
  private tokens: number;
  private readonly maxTokens: number;
  private readonly ratio: number;

  constructor(maxTokens = 10, ratio = 0.1) {
    this.maxTokens = maxTokens;
    this.ratio = ratio;
    this.tokens = maxTokens;
  }

  recordSuccess(): void {
    this.tokens = Math.min(this.maxTokens, this.tokens + this.ratio);
  }

  recordFailure(): void {
    this.tokens = Math.max(0, this.tokens - 1);
  }

  allowsRetry(): boolean {
    return this.tokens > this.maxTokens / 2;
  }
}

export interface RetryOptions {
  budget: RetryBudget;
  maxAttempts?: number; // total attempts, including the first one
  baseDelayMs?: number;
  maxDelayMs?: number;
  deadlineMs?: number; // whole-call deadline, shared by every attempt
}

function parseRetryAfterMs(header: string | null): number | undefined {
  if (header === null) return undefined;
  const seconds = Number(header);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);
  const date = Date.parse(header); // Retry-After may also be an HTTP date
  return Number.isNaN(date) ? undefined : Math.max(0, date - Date.now());
}

function isRetrySafe(init: RequestInit): boolean {
  const method = (init.method ?? "GET").toUpperCase();
  const headers = new Headers(init.headers);
  return IDEMPOTENT_METHODS.has(method) || headers.has("Idempotency-Key");
}

The request loop uses those pieces. Note that the remaining time, not a fixed per-attempt value, sets each attempt’s timeout:

export async function fetchWithRetry(
  url: string,
  init: RequestInit,
  { budget, maxAttempts = 3, baseDelayMs = 100, maxDelayMs = 5_000, deadlineMs = 10_000 }: RetryOptions,
): Promise<Response> {
  const deadline = Date.now() + deadlineMs;

  for (let attempt = 1; ; attempt++) {
    let response: Response | undefined;
    let failure: unknown;
    try {
      const remainingMs = Math.max(1, deadline - Date.now());
      response = await fetch(url, { ...init, signal: AbortSignal.timeout(remainingMs) });
    } catch (err) {
      failure = err; // network error or deadline hit
    }

    if (response && !RETRYABLE_STATUS.has(response.status)) {
      budget.recordSuccess();
      return response;
    }
    budget.recordFailure();

    const retryAfterMs = parseRetryAfterMs(response?.headers.get("retry-after") ?? null);
    const backoffMs = Math.random() * Math.min(maxDelayMs, baseDelayMs * 2 ** (attempt - 1));
    const delayMs = retryAfterMs ?? backoffMs;

    const giveUp =
      attempt >= maxAttempts ||
      Date.now() + delayMs >= deadline ||
      !budget.allowsRetry() ||
      !isRetrySafe(init);

    if (giveUp) {
      if (response) return response; // caller sees the real final status
      throw failure;
    }
    await response?.body?.cancel(); // release the connection before waiting
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }
}

The giveUp check is where the storm gets stopped. Specifically, a retry happens only if attempts remain, the wait fits inside the deadline, the budget allows it, and the request is safe to repeat. Create one RetryBudget per downstream service and share it across all calls to that service, since a budget per call limits nothing.

Stage 5: Retrying Errors That Mean Stop

Not every failure is a request for another try. Some errors are the server asking clients to back off, and retrying them is the opposite of what it asked for.

A 429 or 503 with a Retry-After header is the clearest case. RFC 9110 defines Retry-After as how long the client ought to wait before making a follow-up request, as either seconds or an HTTP date. A client that ignores it and retries on its own schedule is sending load into a service that explicitly said it has none to spare. gRPC has an equivalent: the A6 retry design lets servers send grpc-retry-pushback-ms, and a negative value tells the client not to retry at all.

Non-Idempotent Writes Are Not Retryable by Default

The second case is a write that may have succeeded. If a POST /payments times out, the client cannot tell whether the server never saw it or charged the card and lost the response. Retrying it blindly is how customers get charged twice. That is why the helper above refuses to retry methods outside GET, HEAD, OPTIONS, PUT and DELETE unless the request carries an Idempotency-Key. Similarly, axios-retry only retries 5xx responses on idempotent methods by default.

The fix is to make writes idempotent on the server, usually by storing a client-supplied key with the result and returning the stored result on a repeat. Once the server guarantees that, retrying a POST is as safe as retrying a GET. Until it does, a POST that failed should surface to a human or a reconciliation job, not loop.

Errors That Will Never Succeed

Finally, a 400, 401, 403 or 404 will return the same answer on every attempt. Retrying them wastes capacity and delays the error message the user actually needs. The is_transient predicate in the Python example exists for this reason. Without it, tenacity retries every exception, including the ones caused by a bug in the request.

Stage 6: The Recovery That Never Arrives

Every fix so far reduces the multiplier. None of them caps the total. Even a well-behaved client with three jittered attempts can triple its load on a service that is failing every request, and triple load on a service that is already down is enough to keep it down. This is the metastable state from earlier: the trigger has passed, but the retry volume alone holds the service over capacity.

The fix is a retry budget, which limits retries as a fraction of total traffic instead of per request. When most requests succeed, the budget is full and retries happen normally. When most requests fail, the budget drains, retries stop, and the client fails fast. As a result, load on the struggling service drops back to roughly the original request rate, which gives it room to recover.

How Retry Budgets Work in Real Systems

Every major ecosystem has converged on some version of this idea:

Google SRE: a per-client retry budget where a request is retried only while the ratio of retries to requests is below 10%, which the book says reduces worst-case growth to 1.1x.
gRPC: retry throttling per server name, where each failure subtracts 1 token, each success adds tokenRatio, and retries are disabled once the count falls to half of maxTokens. The TypeScript RetryBudget above follows this design.
Envoy: a retry_budget on the cluster circuit breaker, which caps concurrent retries at budget_percent of active and pending requests. The Envoy API reference gives defaults of 20% with a min_retry_concurrency of 3, and the budget overrides the fixed max_retries threshold when set.
AWS SDKs: a retry quota token bucket in standard mode, which stops retries when failures drain it.

Enabling the Envoy budget takes a few lines on the cluster definition:

clusters:
  - name: inventory
    circuit_breakers:
      thresholds:
        - priority: DEFAULT
          retry_budget:
            budget_percent:
              value: 20.0
            min_retry_concurrency: 3

The AWS SDK Retry Change Arriving in November 2026

The AWS SDK quota is changing, and the timing makes it worth checking now. An AWS announcement from May 2026 describes updated retry behavior that becomes the default in November 2026, opt-in today with AWS_NEW_RETRIES_2026=true. Under the new behavior, the budget holds 500 tokens, a transient retry costs 14 tokens instead of 5, and a throttling retry costs 5. Transient errors back off from a 50 ms base and throttling errors from 1,000 ms, with a 20-second cap.

The higher transient cost means the quota drains faster during a real outage. According to the AWS SDK retry reference, with default settings the budget begins to drain when more than about 22% of requests hit sustained transient failures, or about 32% for throttling. In other words, your AWS clients will stop retrying sooner during a regional problem after November, which is the intended effect. If an alarm or dashboard assumes the old retry counts, test with the flag set before the default flips.

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";

// One client per process: the retry quota lives on the client instance
export const dynamo = new DynamoDBClient({
  region: "eu-central-1",
  retryMode: "standard",
  maxAttempts: 4, // DynamoDB's own default under the 2026 behavior
});

The comment on that client is the important line. The reference notes that the token budget is typically scoped to a single SDK client instance. Consequently, code that builds a new client per request gets a fresh, full budget every time, which disables the protection entirely.

Real-World Scenario: A Checkout Path During a Database Failover

Consider a mid-sized e-commerce backend run by a small platform team, with a checkout flow that crosses four hops: a browser, a backend-for-frontend, an order service and an inventory service in front of Postgres. All services run in an Istio mesh. Over the past year, different engineers added retries at different layers, each for a sensible local reason. The BFF uses axios-retry with three retries. The inventory service wraps its database calls in tenacity with five attempts. Nobody touched the mesh defaults.

During a routine managed-database failover, Postgres refuses connections for a short window. Inventory requests start failing, so tenacity retries each one five times. Meanwhile, some inventory pods saturate their worker pools and begin refusing connections, which triggers the sidecar’s default connect-failure retries from the order service. Above that, the BFF sees 503s from the order service and retries each request three more times. In the worst case, a single checkout click becomes 4 × 3 × 5 = 60 database attempts.

When Postgres comes back, it faces tens of times its normal connection rate. Each burst of reconnects causes timeouts, the timeouts cause more retries, and the database tips over again, which is the reconnect storm described in the Postgres connection limit failure modes post. The failover itself is short. The incident lasts until someone turns retries off or sheds traffic at the edge.

Fixing this scenario is mostly deletion. Inventory keeps its retries, cut to three attempts with jitter and a shared budget, because it sits closest to the database. Next, the order service disables mesh retries toward inventory. Finally, the BFF drops axios-retry for writes and keeps one jittered retry for idempotent reads. Admittedly, the trade-off is real: during brief blips, a few more requests now fail outright instead of being retried into success. That small, bounded error rate replaces an unbounded outage.

When to Use Automatic Retries

The failure is plausibly transient: connection resets, refused connections, 502, 503 and 504
The operation is idempotent, either by HTTP method or through an idempotency key the server enforces
Your layer is the one closest to the failing dependency and owns the retry for that hop
The total time, including every backoff, fits inside the caller’s deadline
A retry budget or quota caps retry volume as a share of traffic

When NOT to Use Automatic Retries

A layer below you already retries the same operation, which multiplies load instead of adding resilience
The server returned 429 or 503 with Retry-After and your client cannot honor the delay
The request is a non-idempotent write with no idempotency key
The error is a 4xx that will fail identically on every attempt
The caller has already timed out, so any retry does work nobody will read

Common Mistakes That Cause Retry Storms

Shipping library defaults, such as tenacity’s infinite immediate retries or Polly’s constant delay without jitter
Adding a retry decorator around an SDK call that already retries internally
Leaving service mesh retries on without knowing they exist
Setting a fixed per-attempt timeout so that three attempts take three times longer than the caller waits
Creating a new SDK client per request, which resets the AWS retry quota on every call
Retrying without logging, so the retry count only shows up in the postmortem
Using a circuit breaker as the only defense; breakers open after the storm starts, while budgets prevent it

That last point deserves a note. A circuit breaker and a retry budget solve different problems, and the circuit breaker and resilience patterns guide covers where breakers fit. Use both, but do not expect a breaker to fix a policy that multiplies load.

How Do You Know a Retry Storm Is Happening?

The clearest signal is the ratio of attempts to logical requests. In a healthy system it sits close to 1.0. During retry storms, it climbs toward the product of your retry layers, while the rate of successful responses falls. If you only track one retry metric, track that ratio per downstream dependency.

Other signals that point the same way:

Request rate rises while user traffic is flat. Incoming requests at the struggling service exceed the edge request rate by a widening margin.
Envoy retry counters climb. upstream_rq_retry rising shows retries happening, and upstream_rq_retry_overflow rising shows a budget or circuit breaker refusing them.
Error rate does not fall after the trigger is fixed. If the database is healthy again but errors persist, the load is coming from clients, not the original cause.
Latency sits at exactly the timeout value. A wall of requests at the timeout boundary means work is being abandoned and probably repeated.

These metrics only help if they are already on a dashboard. The Prometheus and Grafana monitoring guide for microservices covers how to wire per-dependency request and retry counters into alerts.

Retry Storms and Rate Limits Are Two Sides of One Contract

Server-side rate limiting and client-side retry policy are the same agreement seen from both ends. A server that sheds load with 429 and Retry-After only helps if clients wait as asked. Likewise, a client with a perfect retry budget still benefits from a server that rejects excess work cheaply instead of queueing it. The rate limiting strategies guide covers the server side, and the same token bucket idea appears on both ends. For LLM APIs specifically, where 429s are routine, the LLM rate limiting and retry strategies guide goes deeper on Retry-After handling.

Conclusion

Retry storms are rarely caused by one bad policy. Instead, they come from several reasonable ones stacked together: a default with no backoff, backoff with no jitter, retries at every layer, timeouts that restart the clock, and no cap on total retry volume. Each looks harmless in isolation, and together they turn a twenty-second blip into a long outage.

The practical takeaway is an audit you can do today. Pick your most critical request path, list every layer that retries, including SDKs and the service mesh, and multiply the attempt counts. If the product is larger than about three, choose one layer to own retries, give it jitter, a shared deadline and a retry budget, and turn the others off. Then put the attempts-per-request ratio on a dashboard so the next storm shows up while it is still small.