The context: 120M users, 99.99% SLA, zero tolerance

The customer ran a 23-year-old monolithic telecom CRM/billing platform. ~480M events a day, 38k peak RPS, p99 hard cap of 250ms. Everything ran on one Postgres with two-phase commit. The problem wasn't the database — it was that XA transactions failed in the real world the moment we added more services.

Why saga, why outbox immediately?

The first instinct is just saga. But adding saga without an outbox is borrowing the most expensive debt in distributed land: lost events, double-processed payments, unreplayable state transitions. Use a transactional outbox.

The DB write and the event publish must be in the same transaction.
The outbox table is drained to Kafka via polling/CDC.
The consumer side guarantees idempotency via a transactional inbox.

Core schema

CREATE TABLE outbox (
  id           UUID PRIMARY KEY,
  aggregate    TEXT NOT NULL,
  type         TEXT NOT NULL,
  payload      JSONB NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at TIMESTAMPTZ
);
CREATE INDEX outbox_unpub_idx ON outbox (created_at)
  WHERE published_at IS NULL;

Saga coordination: orchestration or choreography?

We started with choreography (every service listens for its own events). Three months in, it became undebuggable — nobody knew who triggered what. We switched to orchestration: one saga coordinator drives steps and compensations.

Choreography is gold for small domains, but past 12+ services it turns into trace hell. Orchestration centralizes failure but you gain observability.

Cutover: canary, shadow and big bang

Weeks 1–2: shadow write — write to the new system in parallel, read from the old, diff metrics.
Weeks 3–6: read split — canary 1%, 5%, 25%, 50% of read traffic to the new path.
Weeks 7–10: write canary — region-segmented user cohorts.
Weeks 11–12: remaining cutover and removal of shadow.

Results (real metrics)

p99 latency: 240ms → 78ms
Event drop rate: 0.04% → 0.0001%
Deploy lead time: 4.2h → 11 minutes
Monthly incidents: 18 → 2
Cost: 21% drop month-over-month (from DB consolidation alone)

If we did it again

Skip 2 weeks of choreography — go orchestration first.
Use Debezium from day one instead of an outbox poller.
Keep the idempotency key wider (event-type + aggregate-id + revision).

Article not found

Article not found

More articles