Ledger CutoverBackstage: Ledger Cutover
Month-end reconciliation fell from 3 h 10 min to 14 min, with no downtime in the close window.
Month-end close went from three hours to fourteen minutes, and nobody was paid twice.
Finance now closes the month on the afternoon of the last working day. The nightly batch is switched off, and the payments team no longer keeps a manual duplicate checklist.
The cutover ran account cohort by cohort across eight working days. At no point was the ledger unavailable during a close window.
14min
Month-end reconciliation
Was 3 h 10 min. Measured over three closes.
0
Duplicate payments per quarter
Was 11 in the quarter before cutover.
2.4s
Posting delay, p95
Was up to 24 h under the nightly batch.
The brief
Meridian’s freight invoices were posted by one nightly batch against a vendor-owned database. When the batch failed, nobody knew until morning. When it retried, it paid twice. Finance closed each month by exporting everything to a spreadsheet and matching lines by hand for three hours.
The ask was a ledger that posts invoices as they arrive, reconciles continuously, and can be switched on without a single minute of downtime in the close window.
What shipped
A double-entry ledger service that receives invoice events, posts them per account in order, and reconciles incrementally instead of once a month. Backstage: How the cutover ran.
The old batch ran beside the new ledger for six weeks as a shadow, and every posting was compared before the ledger was allowed to write. Accounts then moved across in eight cohorts.
Prestige, Meridian Freight and every figure on this page are fictional; they demonstrate the theme.
Written from my notes and the pull-request history. Times are working weeks from kickoff. Wrong guesses are left in.
Backstage updated
Step 1 of 6: Problem. The nightly batch was the single source of truth. Invoices posted once a night; every failure waited until morning.
Week 1
A batch job posted every freight invoice once a night, against a database the vendor owned. When it failed, nobody knew until morning. When it retried, it paid twice.
- Duplicate payments per quarter
- 11
- Month-end reconciliation
- 3 h 10 min
- Posting delay
- up to 24 h
The batch had been extended for nine years by three teams. Nobody owned it end to end, and the only monitoring was a morning email that said “OK” even on the nights it had retried.
Step 2 of 6: Constraints. Four limits that were not negotiable. Zero downtime, no schema changes, four people, one assumption.
Week 1
Constraint Limit Type Source Zero downtime in the close window 0 min unavailable, last 3 working days Hard Finance No schema changes on the vendor-owned database Read-only access Hard Vendor contract Team size and language Four engineers, Go only Soft Team Partner payments API is idempotent on reference Retries are safe Assumed Partner docs, unverified The fourth line is still marked ASSUMED. It turned out to be wrong for our own posting call, which is filed as TD-001.
Step 3 of 6: Hypotheses. Three ways the numbers could be wrong. Duplicates, slow matching, ordering: two confirmed, one refuted.
Weeks 2–4
- H1 ✓ Confirmed
Duplicate payments come from batch retries, not from bad source data.
- Test
- Replay 90 days of batch logs against a copy and count payments with identical invoice and amount.
- Evidence
- 11 of 11 duplicates followed a retry of the 02:00 job.
- H2 ✗ Refuted
Reconciliation is slow because of full-table scans.
- Test
- EXPLAIN ANALYZE on the matching query against production-sized data.
- Evidence
- Index scans throughout. 81% of the runtime was the quadratic matching loop.
- H3 ✓ Confirmed
Per-account ordering is enough; global ordering is not needed.
- Test
- Shadow-run for two weeks and diff every posting against the batch result.
- Evidence
- Zero ordering differences across 1.2 M postings. Cross-account transfers excluded, see step 4.
H2 was the guess I was most sure of, and the one that cost the most to disprove. The query plan was clean; the time was going into a nested loop in application code.
The matching loop and its replacement are compared in step 5.
Step 4 of 6: Rejected options. Three things I did not build. A big-bang rewrite, log-based replication, two-phase commit.
Week 4
RejectedBig-bang rewriteNo rollback inside the close window: one bad night would have stopped month-end.
Cost to revisit: High
RejectedLog-based CDC from the legacy databaseThe vendor contract forbids reading the transaction log.
Cost to revisit: High
RejectedTwo-phase commit across ledger and paymentsIt couples the availability of two systems that fail at different times.
Cost to revisit: Medium
Cross-account transfers stayed on the batch until the last cohort, which is why H3 excluded them.
Step 5 of 6: Implementation. A shadow ledger, then a cohort-by-cohort cutover. Six weeks read-only, then eight working days, one cohort at a time.
Weeks 5–11
Decision ledger
- Shadow first
- Six weeks of read-only comparison before any write proved the ledger on real traffic.
- Cohorts of one eighth of accounts
- A bad cohort touches 12.5% of accounts, and can be moved back the same day.
- Idempotency key on every posting
- Added after cohort 3, see TD-001.
The matching loop was the slow part, so it was rewritten as a single set-based query before the shadow run started.
Before: nested loop in app code
-- Called once per open invoice from the Go matcher SELECT id, amount, reference FROM payments WHERE account_id = $1 AND status = 'open' AND amount = $2 AND reference = $3 AND posted_at >= $4 ORDER BY posted_at LIMIT 1;After: set-based match
-- One pass per account cohort INSERT INTO matches (invoice_id, payment_id) SELECT DISTINCT ON (i.id) i.id, p.id FROM invoices i JOIN payments p USING (account_id, amount, reference) WHERE i.cohort = $1 AND p.status = 'open' ORDER BY i.id, p.posted_at;The same match, written as a loop and as one query. Before
After
October close, same data, two systems: 212 lines to check by hand became 3. Step 6 of 6: Measured result. What the close looks like now. Reconciliation 190 → 14 min; duplicate payments 11 → 0.
Weeks 12–14, then three closes
Month-end reconciliation: before 3 h 10 min, after 14 min, 93% lower.
Duplicate payments per quarter: before 11, after 0, 100% lower.
Posting delay, p95: before 24 h, after 2.4 s, 100% lower.
Verdict The shadow ledger earned the cutover; the matching rewrite earned the speed.
What this does not show: Three closes is a small sample, and December's year-end peak is not in it.
Measured from the ledger’s own audit table, not from a dashboard I built for the purpose: close-to-close timestamps for August, September and October 2025.
Trapdoor
The cohort that posted twice
- What
- Cohort 3 posted 212 ledger lines twice for nine minutes during the cutover.
- Cause
- A consumer replayed an offset after a partition rebalance, and the posting call was not idempotent.
- Cost
- Three hours of cleanup and reconciliation. No money moved: payments were still gated by the batch.
- Changed
- Every posting carries an idempotency key. Each cohort starts with a replay drill on a copy.
Cohort 3 was the first cohort with live card-settlement accounts. At 10:42 the reconciliation view showed 212 lines posted twice. The partition holding those accounts had rebalanced; the consumer resumed from the last committed offset, and the posting call, which I had assumed was idempotent because the partner’s was, was not. Reconciliation caught it nine minutes later. No money moved, because payments were still gated by the batch, but three hours went on cleanup and on proving that to Finance.
What changed: every posting now carries an idempotency key (account, source event id), and each cohort starts with a replay drill on a copy of its own data. The assumption in step 2 of the walkthrough, “partner API is idempotent”, is still marked ASSUMED, now with a link here.
This failure, like the rest of the demo site, is fictional.
Want the walkthrough?
Backstage is one toggle away on every case study. If it isn't written up, ask.
hello@prestige-demo.example.org
Booking from March 2027