Skip to main content

Failover & circuit breakers

Providers fail. The point of Nerve is that your code does not have to care.

Two providers per live channel​

ChannelTried firstTried second
SMSTermiiInfobip
Email (CRITICAL/HIGH)PostmarkSendGrid
Email (NORMAL/LOW)SendGridPostmark

If the first provider fails, the second is tried immediately, within the same dispatch. You see one event and one delivery receipt, from whichever provider succeeded.

If both fail, the event is recorded as failed. There is no cross-channel fallback — a failed SMS does not become an email.

One breaker per provider​

Each provider has its own circuit breaker, so an outage is contained:

  • Termii down → the Termii breaker opens → SMS continues via Infobip
  • Postmark down → the Postmark breaker opens → email continues via SendGrid, including for CRITICAL mail
  • Termii down does not affect email at all

When a breaker opens​

A breaker tracks the ratio of failed to total attempts. It opens when:

  • more than 10 attempts have been recorded, and
  • the failure ratio has reached the configured threshold (CB_THRESHOLD_RATIO, default 0.2 — one failure in five)

The ten-attempt floor is what stops a single early failure from pulling a healthy provider out of rotation.

When it closes again​

An open breaker holds its provider out for a cool-off period (CB_COOLDOWN_SECONDS, default 30 seconds). The next request after that is allowed through and the counters reset: a success closes the breaker, a failure re-opens it for another cool-off.

While a breaker is open, calls to that provider are refused without a network round trip, so a dead provider costs you nothing but the failover hop.

A rejection counts as a failure​

The breaker treats two things as failures:

  • a transport error — timeout, DNS, connection refused
  • a successful HTTP call that the provider rejected

Termii, for instance, signals failure by omitting message_id from a 200 response. That counts. A provider that cheerfully returns 200 while dropping every message will still trip its breaker.

What you should do​

Very little. Failover and breaking are entirely server-side.

What is worth doing:

  • Watch delivery receipts, not just accept codes. A 202 tells you nothing about provider health. See Delivery receipts.
  • Do not implement your own provider retry. Retrying a 202-ed event creates a duplicate; Nerve has already tried both providers.
  • Do retry on 5xx from the gateway itself, with the same Idempotency-Key — that is a Nerve-side failure before the event was durable.

Next​