What Is a Consumer Group? A consumer group is a set of consumer instances that jointly consume a topic. Kafka assigns each partition to exactly one consumer within the group at a time. This is what enables parallel processing: multiple consumers in the same group read different partitions simultaneously. flowchart LR subgraph Topic["Topic: orders — 4 partitions"] P0["Partition 0"] P1["Partition 1"] P2["Partition 2"] P3["Partition 3"] end subgraph CG["Consumer Group: inventory-service"] C1["
Continue reading »Offsets
3 posts in this section
Offset Management: Auto-Commit vs Manual Acknowledgment
Why Offset Management Matters The committed offset determines what happens when a consumer restarts. If the offset is committed too early, a crash before processing completes means events are lost. If it is committed too late, a crash after processing but before committing means events are re-processed. flowchart TD subgraph TooEarly["Commit too early → Data Loss"] E1["Commit offset 43"] --> E2["Process record 42"] --> E3["Crash!"] E4["Restart: resume from 43"] --> E5["
Continue reading »Seeking to Specific Offsets: Replay, Recovery, and Time-Based Seeking
Why Seek Instead of Reset? Offset management (auto-commit vs manual acknowledgment) controls when offsets advance during normal processing. Seeking is different: it lets you reposition the consumer to any offset — past or future — programmatically, without touching the committed offset in __consumer_offsets. Common scenarios: Replay from the beginning — reprocess all historical events after a bug fix Resume from a known-good offset — skip a poison pill that’s blocking the consumer Time-based replay — reprocess everything since yesterday 09:00 Startup positioning — always start from the end, ignoring backlog on first launch How Kafka Seeking Works flowchart LR subgraph Broker["
Continue reading »