The Gap Between Saving and Announcing
Save the record, then publish the event, and a crash in between means nobody hears about it. The outbox pattern closes the gap, and the price is at-least-once delivery your handlers have to survive.
Some work doesn’t belong inside a request. Once you move it out, slices start talking to each other through events, a background worker picks up jobs, and a new class of bug shows up that the request/response world never had.
The gap between saving and announcing
A handler saves the ticket, then publishes TicketClosed.
If the process dies between the two, the ticket is closed and nobody hears about it. If the event handler fails, the caller gets an error for a close that actually happened. In-process events can’t fix that. The save and the announcement are two operations, and you can’t make them one by ordering them carefully.
The outbox pattern does fix it. In the same database transaction that saves the ticket, write the event to an outbox table. A background worker reads the outbox and publishes each event, marking it sent. Either both the change and the event are saved, or neither is.
The cost is that delivery becomes at-least-once.
At-least-once means handlers must be idempotent
Every real queue delivers at least once, so every handler eventually sees a duplicate. Not might. Will.
Design for it:
- Check whether the work is already done, and resume rather than repeat.
- Use a natural key (one closing comment per ticket close).
- Record processed message IDs.
Background services in a fleet
Run three instances and you have three schedulers escalating the same overdue ticket three times, and three in-memory job queues that can’t see each other’s jobs.
In production: a shared queue (Service Bus) for jobs. For scheduled work, either a single dedicated worker, a distributed lock, or a scheduler built for it (Azure Functions timer triggers, Hangfire, Quartz.NET).
BackgroundService works perfectly on your machine because your machine is a fleet of one.
Testing time
If scheduled logic reads DateTimeOffset.UtcNow directly, your tests end up relying on seed data that happens to be old enough. Inject .NET’s TimeProvider (and FakeTimeProvider in tests) and a test can move the clock forward an hour instead of hoping.
The point
Moving work out of the request is the easy half. The hard half is everything that used to be implicit: the save and the event happening together, a message arriving once, a scheduler running once. None of those are true by default anymore. Make each one true on purpose.