Architecture · Oct 2026 · 7 min read · 05/05
Beating the PostgreSQL write wall with Kafka
100,000 writes per second hit a physical disk limit. Why we buffered payments in Kafka, and the outbox and idempotency patterns that make it safe.

Working on a large-scale event ticketing system, we hit a problem that's less about code and more about physics.
When a popular event's tickets go live, we've seen write demand spike past 100,000 records per second. PostgreSQL flushes the WAL (write-ahead log) to disk on every transaction commit, an fsync call. Our SSDs cap out under 100,000 IOPS. That's not a tuning problem, it's a physical wall. There's a hard ceiling on how many individual commits per second the disk can physically fsync.
Option 1: Relax durability settings
If you want PostgreSQL to spend less time waiting for data to be written to disk, you can adjust a few settings:
synchronous_commit = off: PostgreSQL can return a successful response without waiting for the WAL data to be flushed to disk. The WAL writer handles the flush in the background, usually about every 200 ms.
wal_compression = on: PostgreSQL compresses WAL data before writing it, which reduces the amount of data that needs to be written to disk.
wal_buffers = 64MB: gives PostgreSQL more memory to hold WAL data before it needs to write it to disk.
It's important to understand that synchronous_commit = off is not the same as fsync = off.
fsync = off disables PostgreSQL's normal disk flushing. If the server crashes, this can lead to data corruption.
synchronous_commit = off is safer. PostgreSQL can confirm a transaction before its WAL data is fully written to disk. If the server crashes, you may lose the most recent ~200 ms of committed transactions, but the database itself should not be corrupted.
For ticket issuance tied to real payments, that wasn't a risk we could accept. We can't risk a customer being charged with no ticket ever issued.
Option 2: Sharding
Sharding means spreading database writes across multiple PostgreSQL servers instead of sending everything to one server.
PostgreSQL does not support sharding natively. Tools like Citus can add this capability, while table partitioning can help organize data within a single server, but partitioning is not the same as sharding.
Sharding can improve write capacity, but it also adds significant complexity. Things like foreign keys, joins, and transactions across multiple servers become much harder to manage correctly.
What we actually did: Kafka as a write buffer
Instead of writing to PostgreSQL for every payment, we first send each payment event to Kafka. Consumers then read these events and write them to the database in batches.
This changes the workload from thousands of individual database commits per second to a much smaller number of batched transactions.
Customers receive their ticket by email and see it in their profile once the database write is complete, usually within a few seconds. For us, that small delay was acceptable because it gave us better performance without sacrificing data safety.
The catch: asynchronous writes
Asynchronous writes introduce a different data consistency problem. There are two failure modes to design for.
Dual write risk
If you save the payment to PostgreSQL and then separately publish the Kafka event, a crash between those two steps can leave you with a successful payment but no event, and no ticket issued.
The fix is the transactional outbox pattern: save the payment and the event in the same database transaction. Then, a separate process reads the outbox and publishes the event to Kafka reliably.
Duplicate processing
Kafka guarantees at-least-once delivery, which means the same event can be delivered more than once.
For example, the consumer can successfully save the ticket to the database, but crash before committing the Kafka offset. Kafka then sends the same event again, and the consumer processes it a second time.
A database transaction alone doesn't prevent this. Both attempts can be valid, successful transactions.
The fix is idempotency at the business level, for example adding a unique constraint on order_id or using ON CONFLICT DO NOTHING. This makes issuing a ticket for the same order more than once a no-op, regardless of how many times Kafka delivers the event.
Moving writes off the critical path solves the throughput problem, but it introduces a new responsibility. You're trading "the database guarantees consistency for you" for "you now have to design consistency explicitly."