Readiness Checks That Don't Turn a Slowdown Into an Outage
A health check that fails on slowness is powerful and easy to overdo. If every instance reports unready at once, the load balancer has nowhere to send traffic and your probe caused the outage.
When something goes wrong in production, you have three questions: is it working (health), where did the time go (traces), and what happened (logs).
The demo version answers none of them. /health says “Healthy” while the database is down, nothing records what a request did, and the logs are sentences you can only grep. Fixing that is mostly wiring. A few decisions inside it are worth slowing down for.
Health checks: probe or SLO?
A health check that only asks “can I reach the store?” misses the case where the store answers in 9 seconds. So you add a latency threshold, and the check fails on slowness, not just errors. That is close to an SLO check, and it is powerful.
It is also easy to overdo.
If a slow dependency makes every instance report unready at once, the load balancer has nowhere to send traffic. A slowdown becomes an outage, and your health check caused it.
Keep readiness about “can this instance serve at all.” Let alerts handle “is it serving well.”
Metrics, the third signal
Traces explain one request. Logs explain one event. Metrics answer “how is it going overall?”: requests per second, error rate, p95 latency, cache hit ratio.
OpenTelemetry handles them the same way as traces (.WithMetrics(m => m.AddAspNetCoreInstrumentation())), and a custom Meter can count business events or cache hits. Alerts are usually built on metrics. Then you use traces and logs to investigate.
What to log
- Log events with IDs, not narratives.
TicketClosed {TicketId} {UserId}beats a paragraph. You can query the first one. - Never log secrets or personal data: tokens, API keys, passwords, full request bodies.
- Pick levels deliberately. Information for business events, Warning for things that recovered, Error for things that didn’t. If everything is Information, nothing is.
- Sample traces in production. Recording every span of every request gets expensive. Head sampling (keep 10%) and tail sampling (keep every error and every slow request) are both standard.
The point
Observability is not “turn on logging.” Health tells the platform whether to send traffic, metrics tell you something is off, traces and logs tell you why. Keep each one doing its own job, and don’t let your readiness probe take the whole fleet down over a slow query.