Three recurring problems across five companies, and how each one got solved structurally — not by asking people to be careful.
Same settlement run, before and after re-architecting the engine for parallel processing at Verifone. The bar is to scale.
The settlement engine ran as a single sequential pass over every batch. Re-architecting it for parallel processing — partitioning batches and running them concurrently with isolated failure domains — took the run from an 8-hour SLA that ate the overnight window to 45 minutes with headroom to spare.
The settlement engine's output to the acquirer is a strict 80-byte fixed-record file. One duplicate transaction in that file used to mean the acquirer rejected it outright — holding funds for every merchant in the batch, not just the duplicate one. The uniqueness key is configurable per settlement service, so a service that needs a different combination of fields to define "duplicate" can set that up independently. Toggle the fix on and off.
With the check active, only unique transactions reach the writer — the file stays clean and the acquirer accepts it on time.
This runs underneath multi-million-transaction settlement services. Since it was introduced, the acquirer file has had zero rejections.
Four companies, one obsession: take bespoke engineering out of onboarding. On the left, integration as a project. On the right, as a socket. Tap a company.
TSYS — configurable card-application forms let banks self-serve their own onboarding.
A request goes Account Service → Processor Gateway, which asks an account-specific singleton actor whether this account was refreshed in the last 15 minutes. Fresh, and it's served straight from Elasticsearch. Stale, and the Gateway calls the processor APIs (Fiserv/Stifel Optis over REST, TSYS TS1/TS2 over SOAP), publishes the refresh event onto Pulsar, and re-indexes Elasticsearch. When an update comes through, the data and the singleton's refresh clock update together — no window where they can disagree. If the processor call errors, whatever's currently in Elasticsearch is served anyway — flagged stale, with a last-updated timestamp for the caller. Pick a scenario.
▷ Fresh — account was refreshed within the last 15 minutes; served straight from Elasticsearch.
Not to be confused with the file-level check in Fig 02 or the account-refresh actor in Fig 04 — this is a third use of the same pattern. When the Processor Gateway fetches transactions from the processor APIs (Optis, TS1/TS2), a duplicate fetch can happen two ways: the same user triggers the fetch more than once, or Directory publishes the same account event onto Pulsar twice. Either way, without a check, the same transaction gets processed twice. A cluster singleton actor sits in front of publish/store via the Gateway — so it only ever happens once. Built at SpendLabs, carried forward into Fiserv. Toggle the fix.
With the singleton active, exactly one pod processes each transaction — Pulsar and Elasticsearch each see it exactly once.
Across 30+ services at Verifone, the testing strategy shifted from unit-test-heavy coverage to container-backed integration tests. A container-backed test spins up a real dependency — a real Postgres, a real broker — instead of mocking it, so a broken contract between two services fails in CI, not in production.
Mocked dependencies pass in isolation. Integration breaks surface after merge — sometimes after deploy.
Real dependencies via Testcontainers. A broken contract fails the build, not the customer.