Transient vs Permanent Failures Not every exception is worth retrying. Retrying a NullPointerException or a schema validation error wastes time and delays other records. Retrying a database timeout or a downstream HTTP 503 is exactly right — the error is temporary and will likely resolve. flowchart TD Ex["Exception in listener"] Q{"Transient?\n(DB timeout, HTTP 503,\nnetwork blip)"} Q -->|Yes| Retry["Retry with BackOff"] Q -->|No| Skip["Call recoverer immediately\n(no retries wasted)"] Retry -->|"still failing after\nmax retries"

Why Seek Instead of Reset? Offset management (auto-commit vs manual acknowledgment) controls when offsets advance during normal processing. Seeking is different: it lets you reposition the consumer to any offset — past or future — programmatically, without touching the committed offset in __consumer_offsets. Common scenarios: Replay from the beginning — reprocess all historical events after a bug fix Resume from a known-good offset — skip a poison pill that’s blocking the consumer Time-based replay — reprocess everything since yesterday 09:00 Startup positioning — always start from the end, ignoring backlog on first launch How Kafka Seeking Works flowchart LR subgraph Broker["

Why Partitioning Strategy Matters How you route messages to partitions determines: Ordering: only messages in the same partition are ordered relative to each other Parallelism: how evenly work is distributed across consumers Hot spots: if one key generates 90% of traffic, one partition (and one consumer) gets 90% of the load flowchart TD subgraph Routing["Message Routing Decision"] Msg["Message"] HasKey{Has key?} HasPartition{Explicit partition?} KeyHash["hash(key) % numPartitions\n→ deterministic, same partition always"] RoundRobin["Sticky partitioning\n(batch to same partition,\nthen round-robin)"

Before You Ship This is the checklist distilled from everything in this series. Work through it before your first production deployment. Each item links to the article where it’s covered in depth. Producer Checklist Durability # Never lose data on leader failure spring.kafka.producer.acks=all # At least 2 brokers must acknowledge every write spring.kafka.producer.properties.min.insync.replicas=2 # Enables exactly-once message delivery (required for transactions) spring.kafka.producer.properties.enable.idempotence=true Do: Set acks=all and min.insync.replicas=2 for any topic that carries business data.

Prerequisites Docker Desktop installed and running docker compose v2 (bundled with Docker Desktop 4.x+) Ports 9092, 9093, 9094 free on your machine All articles in this series assume a running local Kafka cluster. Start with the single-broker setup for articles 1–6, then switch to the 3-broker cluster when we cover replication and fault tolerance. Single-Broker Cluster (Development) This is the simplest setup — one Kafka node running in combined mode (broker + controller).

Two Testing Strategies Strategy Speed Fidelity Use when @EmbeddedKafka Fast (~2s startup) In-process broker, not 100% identical Unit/integration tests — CI fast path KafkaContainer (Testcontainers) Slower (~10s startup) Real Kafka broker in Docker Acceptance tests, DLT/transaction validation Use both: @EmbeddedKafka for the bulk of tests, KafkaContainer for the smoke suite that validates real-broker behaviour. Test Dependencies <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.testcontainers</groupId> <artifactId>kafka</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.awaitility</groupId> <artifactId>awaitility</artifactId> <scope>test</scope> </dependency> @EmbeddedKafka — Fast Integration Tests Testing a Producer @SpringBootTest @EmbeddedKafka( partitions = 1, topics = {"orders"}, brokerProperties = {"log.

The Problem Kafka Solves Imagine an e-commerce platform. A customer places an order. What needs to happen next? Inventory must be reserved Payment must be charged A confirmation email must be sent The warehouse must be notified to pick and pack Analytics must record the sale Fraud detection must evaluate the transaction One request. Six downstream systems. In a traditional REST architecture, the Order Service calls each of those six services directly — synchronously, one after another.