ACID Is a Contract, Not a Religion: How Real Systems Keep Money Safe
Most engineers learn ACID early. Most engineers also learn later that ACID is not the whole story. That is not a contradiction—it is a scope problem. This article is a practical architecture guide to double-entry ledgers, concurrency control, isolation levels, and the quiet discipline that separates payment systems that work from payment systems that work until they don't.
Table of Contents
Most engineers learn ACID early. Most engineers also learn later that ACID is not the whole story.
That is not a contradiction. It is a scope problem.
ACID is a contract for one database transaction on one database system. Money movement is a contract across time, networks, services, humans, regulators, and attackers. Production-grade correctness lives above the database. It uses the database. It does not worship it.
This article is a practical and theoretical guide. It covers ACID. It covers concurrency. It covers security. And it focuses on the one design that repeatedly wins in finance: the ledger.
I. Invariants First, Tools Second
Correctness means your invariants hold. Not "the database committed."
In payments and banking, typical invariants look like this:
- Money cannot appear or disappear inside your system.
- Every posted transfer is balanced.
- A posted entry is never edited. It is only offset by a later entry.
- A transaction is processed at most once, even if requests repeat.
- "Available funds" never go below policy limits.
- Every balance can be explained from an audit trail.
Notice what is missing. No invariant says "we used serializable isolation." Isolation is a tool. Invariants are the goal. That framing makes ACID easier to reason about. It also makes it easier to admit where ACID ends.
II. What ACID Guarantees -- And Where It Ends
ACID is still essential. But it is narrower than many teams assume.
Atomicity. All writes in a transaction commit or none do. The "no partial update" guarantee. Non-negotiable for ledger posting.
Consistency. Constraints you define are enforced -- foreign keys, checks, unique constraints, triggers. This is "database rules correctness," not "business correctness." The distinction matters more than most architects admit.
Isolation. Concurrent transactions behave as if they ran with some defined interference rules. The rules depend on the isolation level. Many production defaults are weaker than engineers think (Berenson et al., 1995).
Durability. Committed data survives crashes. Within the assumptions of storage and replication.
Now the gap. ACID does not guarantee correctness across multiple services, caches, read replicas, message brokers, retries, timeouts, or partial network failures. It also does not guarantee your business invariants -- not unless you encode them, and encode them in a way concurrency cannot break.
That is the hard part.
Isolation levels are about anomalies, not feelings
The SQL standard defines isolation levels by which anomalies are allowed. The classic paper that clarified the mess is Berenson et al.'s A Critique of ANSI SQL Isolation Levels (1995). Old. Still worth reading. Because the failure modes are timeless.
The anomalies you should fear in money code:
- Lost update: Two debits read the same balance and both succeed.
- Write skew: Two transactions each maintain constraints locally but violate a global invariant together.
- Phantoms: A range query changes under you, breaking "check then act" logic.
- Read skew: A transaction reads related rows from different points in time.
Snapshot isolation (SI) is especially treacherous. It is attractive. It scales. But SI can permit write skew -- two transactions each see a world where a constraint holds, both commit non-conflicting writes, and the global invariant breaks (Berenson et al., 1995).
If you want the simple mental model: serializability. The effect is equivalent to some serial order. PostgreSQL documents this clearly. But serializable costs something. So high-scale systems do not always run "serializable everywhere." They instead make serializability small. They reduce the contention domain.
This is the beginning of ledger thinking.
III. The Ledger Move: Derived State Over Mutable State
Many systems start with an accounts.balance column. Then they do: read balance, check enough funds, write new balance.
This is the concurrency bug factory. It can be patched with locks, retries, careful SQL. But the core model is fragile.
A ledger model flips the direction:
- You store immutable postings.
- You compute balances as a sum of postings.
- You treat the computed result as derived state.
Three properties arrive immediately: an audit trail by default, the ability to rebuild derived state after bugs, and the ability to reason about correctness over time.
A real bank-grade ledger is usually double-entry. This is not tradition. This is algebra. Griffin's "immutable bank" write-up states it plainly: journal entries have line items, and the sum of debits equals the sum of credits (Griffin, 2022). That invariant prevents "lost money." It does not prevent "wrong recipient." But it guarantees conservation inside the system.
TigerBeetle, a modern financial ledger database, expresses the same invariant as a first-class primitive. Different implementation. Same math.
Ledger anatomy: accounts, journal entries, postings
Three core concepts:
- Account: a bucket of value in one currency or asset type.
- Journal entry: a transaction record carrying metadata.
- Posting (line item): a debit or credit applied to an account, linked to a journal entry.
Minimal relational schema:
CREATE TABLE accounts (
account_id BIGINT PRIMARY KEY,
currency CHAR(3) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
closed_at TIMESTAMPTZ
);
CREATE TABLE journal_entries (
entry_id UUID PRIMARY KEY,
idempotency_key TEXT NOT NULL UNIQUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
posted_at TIMESTAMPTZ,
status TEXT NOT NULL CHECK (status IN
('PENDING','POSTED','REVERSED','FAILED')),
type TEXT NOT NULL,
meta JSONB NOT NULL DEFAULT '{}'
);
CREATE TABLE postings (
posting_id BIGSERIAL PRIMARY KEY,
entry_id UUID NOT NULL REFERENCES journal_entries(entry_id),
account_id BIGINT NOT NULL REFERENCES accounts(account_id),
side TEXT NOT NULL CHECK (side IN ('DEBIT','CREDIT')),
amount_minor BIGINT NOT NULL CHECK (amount_minor > 0),
currency CHAR(3) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX postings_account_time ON postings(account_id, created_at);
CREATE INDEX postings_entry ON postings(entry_id);
The invariant: for each POSTED journal entry, per currency, sum(debits) = sum(credits). Enforce this in a posting stored procedure or with deferred constraints. Do not rely on developer discipline alone.
You are no longer protecting "a balance number." You are protecting a conservation law.
IV. Concurrency Control: Making Serializable Small
Posting a journal entry and its postings must be atomic. That is where ACID shines. One transaction: dedupe via idempotency key, validate business rules, append postings, mark entry as posted, emit an outbox event. If it commits, the ledger is updated. If it rolls back, nothing happened.
So far, this is "ACID done right." But concurrency is where systems fall apart.
A global serializable database is expensive. But most money invariants are local -- per account, per pair of accounts, per "available" vs "pending" bucket. So you enforce serial behavior by locking the minimal rows.
Approach A: Deterministic pessimistic locks
When moving value from account A to account B, lock both in sorted order to avoid deadlocks:
BEGIN;
-- Lock order by account_id to prevent deadlocks
SELECT 1 FROM accounts
WHERE account_id = LEAST($from, $to) FOR UPDATE;
SELECT 1 FROM accounts
WHERE account_id = GREATEST($from, $to) FOR UPDATE;
-- Insert entry (idempotent)
INSERT INTO journal_entries(entry_id, idempotency_key,
status, type, meta)
VALUES ($entry_id, $idem, 'PENDING', 'TRANSFER', $meta)
ON CONFLICT (idempotency_key) DO UPDATE
SET entry_id = journal_entries.entry_id
RETURNING entry_id;
-- Append postings (balanced)
INSERT INTO postings(entry_id, account_id, side,
amount_minor, currency)
VALUES
($entry_id, $from, 'DEBIT', $amt, $cur),
($entry_id, $to, 'CREDIT', $amt, $cur);
UPDATE journal_entries
SET status = 'POSTED', posted_at = now()
WHERE entry_id = $entry_id;
COMMIT;
Simple. Powerful. This gives you per-account serial execution even at Read Committed. Row locks are a blunt instrument that prevents write skew. PostgreSQL also offers serializable via SSI -- but even then, plan for retries.
Approach B: Optimistic concurrency with retries
Many distributed databases default to serializable with retries. CockroachDB runs serializable by default and surfaces retry errors. FoundationDB provides strict serializability using optimistic concurrency control.
The cost is operational. Your application must retry safely. Your operations must be idempotent. Your clients must tolerate latency spikes under contention.
Pick one: you lock and you block, or you run optimistic and you retry. Both can be correct. Both have costs. If you cannot retry safely, do not pretend you can. Use locks.
The snapshot isolation trap
If your invariant depends on a predicate -- not a single row -- then snapshot isolation can break it silently. Two transactions each see a world where a constraint holds, both commit non-conflicting writes, and the global invariant shatters.
Three fixes:
- Lock the predicate range.
- Run true serializable.
- Encode the invariant as a single-row "gate" you can lock.
Ledger systems often use the third option. They force invariants into a small lockable domain. Ports & Grittner's SSI paper describes how PostgreSQL detects dangerous structures to prevent anomalies under SERIALIZABLE.
V. Settlement, Holds, and the Two-Phase Pattern
Internal posting should be crisp. External settlement is messy.
Griffin explicitly separates the ledger from a "transactor" layer that tracks states like instruction, held, failed, completed (Griffin, 2022). That pattern is common. It exists because networks fail. Payment rails time out. Counterparties disagree. Humans file disputes.
So you model two distinct things:
- Ledger posting: internal accounting truth.
- Settlement workflow: external reality convergence.
Do not post final ledger entries for external transfers until you know what they mean. Use pending holds instead.
[Internal] [External]
+-----------------------+ +------------------------+
| Ledger (immutable) | | Settlement Workflow |
| - journal_entries |<----->| - INSTRUCTED |
| - postings | | - HELD |
| - account_balances | | - COMPLETED / FAILED |
+-----------------------+ +------------------------+TigerBeetle calls this "pending transfers." Griffin calls it "held" funds. Card networks call it authorization vs capture. Different names. Same concept.
A practical model: available (spendable now) and pending (reserved for an in-flight action). A card authorization moves value from available to pending. A capture moves pending to posted final. A void moves pending back to available.
This has two benefits: you prevent overspend while external settlement is uncertain, and you can expose truthful UX. "Pending" is not a bug. It is honesty.
VI. Beyond the Single Database: Sagas, Outboxes, Idempotency
You can build a distributed two-phase commit system. You can also build a distributed deadlock machine.
Most modern architectures avoid distributed transactions across services. They use local transactions plus workflows.
Pattern 1: Saga
A saga is a sequence of local transactions. Each step commits locally and triggers the next step via messages. If a step fails, compensating transactions execute (Microservices.io, Saga Pattern). Sagas are not "eventual consistency hand-waving." They are explicit. They are auditable. They require strong idempotency discipline.
Pattern 2: Transactional outbox
The dual-write problem is real. You update the DB. You publish to Kafka. One succeeds. One fails. Now what? Confluent describes this failure mode clearly.
The fix is the transactional outbox: in the same DB transaction as your business write, insert an "event" into an outbox table. A separate publisher reads the outbox and publishes reliably. If the DB commits, the message eventually publishes. If the DB rolls back, no message exists.
This turns "distributed atomicity" into "local atomicity + reliable forwarding." It is boring. That is why it works.
Write Path (single DB transaction)
+-----------------------------------------+
| 1. Insert journal_entry |
| 2. Insert balanced postings |
| 3. Update cached balance |
| 4. Insert outbox_event |
| 5. Mark entry POSTED |
+-----------------------------------------+
|
v
Outbox Poller (async)
+-----------------------------------------+
| Read unsent events -> Publish to broker |
| Mark events as sent |
+-----------------------------------------+Idempotency over exactly-once
"Exactly once" end-to-end delivery is rare. Networks duplicate. Clients retry. Brokers redeliver. Timeouts cause ambiguous outcomes.
Build around a stronger and more achievable property: idempotent processing. Require an idempotency key for each client intent. Store it with a unique constraint. Return the prior result on retry.
If you do only one thing after reading this article, do this: put a unique constraint on idempotency_key, treat duplicates as normal, and make your handlers safe to run twice. You will prevent a shocking number of "double debit" incidents.
VII. Read Paths, Projections, and Reconciliation
Writes are often strongly consistent. Reads often are not. If you read from a replica, you can see stale data. Even small lag can violate user expectations. AWS documentation notes replica lag can vary with write load.
For money UX, define a policy: for "balance right now," read from the writer. Or use session-level consistency controls. Or provide "as of timestamp" semantics. Do not silently mix. The worst bug is a bug that looks like fraud.
Performance without lies
A pure ledger says: compute balance by summing postings. Correct. Also expensive at scale. So you add projections:
Layer 1 -- Immutable ledger. Postings are append-only. They are the system of record.
Layer 2 -- Cached balances. Maintain account_balances as a derived table. Update it in the same transaction as postings. Treat it as a cache.
Layer 3 -- Checkpoints. Store periodic snapshots of balances. Daily is common. This speeds up "balance as of time T."
Layer 4 -- Reconciliation. Recompute balances from postings. Compare to cached balances. Alert on drift. Repair by replay.
Reconciliation is not optional in serious systems. It is your safety net. And it is your debugging superpower. Jepsen-style testing and real-world incident history show that databases can behave in surprising ways under faults. Your ledger plus reconciliation is how you detect and recover from those surprises.
VIII. Security: The Threat Model Is Not Just Hackers
In payments, security includes external attackers, internal mistakes, insider abuse, operational accidents, data corruption, and dispute processes. A ledger helps because it is append-only and explainable. But you still need controls.
Least privilege everywhere. Use separate DB roles: a posting role that can insert postings and journal entries, a read role that can query but not mutate, and an admin role that is rare and audited.
Immutability by policy and by mechanism. Do not allow UPDATE or DELETE on postings in normal paths. Use DB permissions to enforce. Griffin explicitly prefers writing a reversing journal entry rather than rewriting history (Griffin, 2022). That is the correct accounting approach. It is also the correct security approach.
Auditability and dispute handling. Fast payments produce disputes. They need processes, timelines, and compensation rules. The RBI has published frameworks for turn-around time and customer compensation for failed transactions. The World Bank has discussed dispute resolution in fast payment systems. These rules push architecture in one direction: you need states, traceability, and proof of what happened. A ledger is your proof substrate.
API security and abuse resistance. Rate limit status checks. Throttle balance refresh. Make endpoints idempotent. Reject abusive retry storms. Correctness and security overlap here. Most fraud starts as a race condition plus social engineering.
IX. The Blueprint: A Ledgered Payment Service
Here is an architecture you can ship.
+-----------+
| API GW |
+-----+-----+
|
+-------v--------+
| Payment Service |
+-------+--------+
|
+----------------+----------------+
| |
+------v-------+ +-------v--------+
| Ledger DB | | Settlement |
| (PostgreSQL) | | Workflows |
| | | |
| accounts | | INSTRUCTED |
| journal_ent |<--------------->| HELD |
| postings | | COMPLETED |
| outbox_evts | | FAILED |
| acct_balances| +----------------+
+--------------+
|
+------v-------+
| Outbox Poller|---> Message Broker ---> Consumers
+--------------+ (idempotent)Write path: internal transfer. Validate input. Begin transaction. Lock accounts in deterministic order. Insert journal entry with idempotency key. Insert balanced postings. Update cached balances. Insert outbox event. Mark entry posted. Commit.
Write path: external payment. Create workflow row. Reserve funds (hold). Call external rail. On confirmation, post final journal entry. On failure, release hold. Always reconcile asynchronously.
Concurrency strategy. Pick one: pessimistic per-account locks for predictable behavior, or serializable + retries if your platform supports it and your code is retry-safe. PostgreSQL's SSI design is well-documented. Distributed databases like Spanner provide even stronger properties, but at significant system complexity. Do not choose based on ideology. Choose based on operational reality.
Why Ledgers Endure
The ledger approach wins because it gives you properties you cannot bolt on later.
Explainability. Every balance change has a reason. You can answer "why did this change" from postings.
Recoverability. Ship a bug? Replay. Lose a cache? Rebuild.
Controlled concurrency. Lock the right things. Isolate invariants.
Safe reversals. Reverse by offsetting, not rewriting. That is exactly what auditors want.
Workflow integration. Holds, settlement states, disputes, and chargebacks become additive layers. They do not contaminate the ledger's truth.
ACID gives you a correct commit boundary. It does not give you correct money. Money safety comes from a double-entry ledger that conserves value, carefully scoped concurrency control, idempotency as a first-class rule, workflow states for external uncertainty, outbox-based event publishing, reconciliation as a normal operation, and security controls that assume mistakes and abuse.
This is the quiet secret of financial software. It is less about heroic algorithms. It is more about disciplined invariants. It is engineering that respects the universe's favorite rule: things fail.
And that is why ledgers endure. Not because they are old. Because they are hard to cheat.
References
- Berenson, H. et al. (1995). A Critique of ANSI SQL Isolation Levels. ACM SIGMOD. PDF
- PostgreSQL Documentation. Transaction Isolation. postgresql.org
- Ports, D. & Grittner, K. (2012). Serializable Snapshot Isolation in PostgreSQL. VLDB. PDF
- Griffin Bank. (2022). Building an Immutable Bank. griffin.com
- TigerBeetle. Financial Transactions Database. tigerbeetle.com
- Richardson, C. Saga Pattern. Microservices.io. microservices.io
- Richardson, C. Transactional Outbox. Microservices.io. microservices.io
- Confluent. Why Dual Writes Are a Bad Idea. confluent.io
- Reserve Bank of India. Turn Around Time and Customer Compensation. rbi.org.in
- World Bank. Fast Payment Systems. worldbank.org
- Kingsbury, K. Jepsen: Distributed Systems Safety Research. jepsen.io
- AWS. Working with Read Replicas. docs.aws.amazon.com
