Published 27 Oct 2026

Retries Multiply: Designing How Your API Fails

Three layers that each retry 3 times turn one click into 27 database calls during an incident. Retry at one layer, fit every budget inside its caller's, and only retry writes that are safe to repeat.

Every dependency fails eventually. The demo version handles that by waiting. A store that hangs for 10 seconds holds the request (and a server thread) for 10 seconds, a single blip becomes a 500, and an outage means every caller waits out every failure.

Resilience is designing how your API fails instead: within a known time budget, retrying only what’s safe, and backing off a dependency that’s down instead of piling on.

Retries multiply

If the client retries 3 times, your API retries the store 3 times, and the store retries its database 3 times, one user’s click becomes 27 database calls during an incident. You built a load amplifier and pointed it at the thing that’s already struggling.

Retry at one layer, usually the one closest to the failure. And make sure every layer’s time budget fits inside its caller’s. A 30 second retry policy behind a 10 second request timeout is just a slower way to fail.

Don’t retry writes that may have happened

A read that timed out is safe to retry. A write that timed out may have happened. Retry it and you might create the thing twice.

So the default is to retry reads and not writes. The better fix is to make writes safe to repeat: the caller sends an Idempotency-Key, and the API returns the original result if it sees the same key twice. Once writes are idempotent, retrying them becomes safe again, on your side and the caller’s.

You don’t have to build this by hand

Calling another API with HttpClient? Microsoft.Extensions.Http.Resilience gives you the whole pipeline in one line:

builder.Services.AddHttpClient<BillingClient>().AddStandardResilienceHandler();

That is a total timeout, retry, circuit breaker, and attempt timeout, in that order. Worth knowing what each piece does, because you will need to tune it.

Tuning the breaker

A breaker that opens too easily turns small blips into outages. One that never opens is decoration.

Start from real traffic. The minimum throughput should be well above what a quiet minute produces, so a handful of errors at 3am doesn’t open it. And alert on breaker state changes. An open circuit is the clearest signal you’ll get that a dependency is in trouble.

What failure should look like

  • A transient blip is invisible to callers.
  • A hung dependency is a 504 inside the budget.
  • A down dependency is a fast 503 with Retry-After.
  • A budget that can’t possibly work refuses to start.

The point

Resilience isn’t “add retries.” It’s deciding how long anything is allowed to take, which layer gets to retry, and what callers see when the answer is no.