Notification Delivery: Queues, Retries and Deadlines

Notifications: a successful API call is not delivery

A notification system has several outcomes: accepted by our service, accepted by a provider, delivered to a device or mailbox, and read by a person. Those are different events. This exercise designs email, SMS, and push delivery without promising that a provider or recipient device will always be reachable.

A burst that a queue cannot solve alone

Assume a campaign has 600,000 recipients and a target completion time of two minutes. That requires 5,000 provider submissions per second. If the account's measured and permitted throughput is only 2,000 per second, the best-case drain time is five minutes, before retries. Adding consumers cannot defeat that constraint.

A queue absorbs a burst; it does not make a missed deadline achievable. Choose among starting earlier, obtaining more capacity, narrowing the audience, or changing the deadline. Treat all rates here as hypothetical. Provider quotas depend on the product, account, endpoint, and destination; there is no universal “Twilio limit” or “Apple limit.”

FCM's guidance discusses throttling, retry behavior, exponential backoff, and smoothing spikes. Use provider instructions to configure each adapter rather than applying the same retry schedule everywhere. Firebase: sending FCM messages at scale.

Record intent before calling the provider

The producer submits an operation ID, recipient, template version, channel, purpose, and expiry. In one transaction, store the accepted notification and an outbox event. Return acceptance only after that transaction commits. A dispatcher publishes outbox events to channel-specific queues and can replay safely.

A worker checks expiry and current preferences, obtains permission from the provider-specific rate limiter, and records an attempt. It then calls the provider and stores the provider's identifier or error. Keep immutable attempt records separate from the notification's current state. This lets an operator distinguish “never attempted” from “attempted, response unknown.”

Reserve capacity for time-sensitive authentication messages. If every consumer and quota token is occupied by a campaign, merely naming a queue “high priority” does not create priority. Allocate concurrency and throughput budgets explicitly.

The timeout that defeats a simple deduplication table

Consider this sequence:

1. The worker sends message N to a provider.
2. The provider accepts N.
3. The response is lost.
4. The worker restarts.

Writing sent=true before step 1 risks suppressing a message that was never sent. Writing it after step 3 risks a duplicate on retry. A local uniqueness constraint cannot make a database write atomic with an unrelated provider API.

Where supported, send the same provider idempotency key for all retries of one logical delivery. Otherwise query the provider using a stable reference when possible, and define an ambiguity policy per channel. A duplicate marketing email may be worse than a missed one; a critical alert may justify a duplicate with the same incident ID. The service must represent that trade-off honestly.

Retry only while the message remains useful

For a two-minute login code, an attempt after five minutes is harmful even if it succeeds. Persist expires_at and check it before each retry. Refreshing the login flow should invalidate the older code according to the authentication service's rules.

Classify failures rather than retrying all errors. A malformed destination or invalid device token usually needs data correction. Temporary throttling needs a delayed retry that follows provider guidance. Use jitter and a retry budget so a provider outage does not produce a synchronized recovery surge. Exhausted attempts go to a review queue with a reason and a safe replay action.

The interactive token bucket above illustrates a local mechanism. A real multi-worker quota needs coordinated enforcement or partitioned token budgets; otherwise ten workers can each spend the full account allowance.

Preferences and receipts are changing state

Check marketing permission at send time because a user can unsubscribe after the campaign was queued. Record suppression as a state, not a silent disappearance. Keep transactional and marketing purposes separate so a preference is applied to the right messages. Localize templates and retain the rendered version or its reproducible inputs for investigation, with a deliberate retention policy for personal data.

Delivery callbacks can be duplicated or arrive out of order. Verify provider signatures, deduplicate event IDs, and validate state transitions. A late “accepted” callback should not overwrite a terminal failure or a delivery receipt without provider-specific reasoning.

Measure the age of the oldest unexpired message, deadline misses, duplicates reported by users, provider acceptance, and confirmed delivery separately. Test a lost provider response, a revoked preference while queued, a throttled campaign, and a callback replay. These reveal whether the service's promises survive the cases that a simple “sent” counter hides.