Kafka in Production: Failure Scenarios for Staff-Level Interviews

A single misconfigured setting in Kafka doesn't just slow a page down. It can silently vaporize financial transactions. In this deep dive we walk through real production failure scenarios and the exact reasoning interviewers look for in staff-level system design interviews. You'll learn: \- Why acks=1 loses data, and the four configuration layers for zero data loss \- Consistency vs availability: why a healthy cluster refuses writes on purpose \- Pushing from 400,000 to 2 million events/s with batching, linger.ms and compression \- How retries reorder a debit and a credit, and how idempotent producers fix it \- Rebalance storms, cooperative sticky assignment and static membership \- Hot partitions and the irreducible trade-off between ordering and scale \- How deleting and recreating a topic silently skips hours of data \- A 3 AM role-play: 85 under-replicated partitions, a disk at 96%, and why you never restart the broker \- Exactly-once with Kafka transactions, and exactly-once into PostgreSQL \- Tiered retry topics and dead letter queues \- Split brain, KRaft quorums and vanishing tombstones \- System design at scale: multi-tenant topics, quotas, event-driven sagas and change data capture with Debezium Ideal for backend and platform engineers preparing for senior and staff-level interviews. Chapters 00:00 Distributed systems have no X-ray 01:08 Why Kafka interviews matter 02:19 acks=1 and the 200 missing payments 03:27 Four layers of zero data loss 05:21 The CAP theorem trade-off 06:31 Throughput: taxis vs buses 07:42 batch.size, linger.ms and compression 09:38 When retries reorder messages 11:38 Idempotent producers 12:48 Message keys and partition ordering 13:12 Flash sale: buffer exhausted 14:22 Decoupling with a local buffer and circuit breaker 16:18 Rebalance storms 18:13 Eager vs cooperative sticky rebalancing 19:25 Static group membership 20:12 Hot partitions and the hot key problem 23:18 Silent data loss after recreating a topic 25:20 auto.offset.reset and topic versioning 26:08 3 AM: 85 under-replicated partitions 28:54 Why you never restart the broker 29:42 Disk at 96%: the 15-minute response 31:36 Throttled reassignment, Cruise Control, tiered storage 33:36 Duplicates in consume-transform-produce 34:47 Kafka transactions 37:11 Exactly-once into PostgreSQL 39:56 Head-of-line blocking from retries 41:08 Tiered retry topics and DLQs 42:42 Split brain 44:38 KRaft and quorum math 45:27 Log compaction tombstones 48:33 Multi-tenant topic design 50:07 Tenant tiers and client quotas 51:40 Event-driven sagas 53:15 Compensating transactions 54:51 Change data capture with Debezium 57:13 Recap 58:46 Will offsets still matter in five years? #Kafka #SystemDesign #DistributedSystems #SoftwareEngineering #InterviewPrep #BackendEngineering Interview Prep Podcast