Why Filter at the Container Level? Multiple consumers can share a topic. The inventory service only cares about PLACED orders; the analytics service wants everything. Rather than putting if (event.getStatus() != PLACED) return; at the top of every listener, Spring Kafka lets you filter records before they reach your method — keeping business logic clean and making the filter reusable across listeners. How RecordFilterStrategy Works flowchart LR Broker -->|"poll() → [r1, r2, r3, r4]"

The Problem: Poison Pills at Deserialization Time A malformed byte sequence — truncated JSON, wrong Avro schema, corrupt payload — throws an exception during deserialization, before the listener method is called. Without special handling, this record blocks the partition indefinitely: the consumer fetches it, fails to deserialize, and fetches it again on the next poll. sequenceDiagram participant Broker participant Container participant Deserializer loop forever without ErrorHandlingDeserializer Container->>Broker: poll() Broker-->>Container: [good-record, CORRUPT-RECORD, good-record] Container->>Deserializer: deserialize(CORRUPT-RECORD) Deserializer-->>Container: JsonProcessingException 💥 Note over Container: Partition blocked — same record on every poll end ErrorHandlingDeserializer solves this by catching the deserialization exception and wrapping it in a DeserializationException that the listener container can route to the error handler.

The Duplicate Problem With acks=all and retries enabled, a produce request might be acknowledged by the broker, but the acknowledgment is lost in the network before reaching the producer. The producer, seeing no response, retries — sending the same record again. The broker writes it a second time. The consumer sees a duplicate. sequenceDiagram participant Producer participant Leader as Broker (Leader) Producer->>Leader: ProduceRequest: OrderPlaced (orderId=1001) Leader->>Leader: Write record at offset 42 ✓ Leader--xProducer: ProduceResponse LOST (network failure) Note over Producer: No ack received — retrying Producer->>Leader: ProduceRequest: OrderPlaced (orderId=1001) [RETRY] Leader->>Leader: Write record at offset 43 ✓ (DUPLICATE!

The Serialization Problem Kafka stores bytes. KafkaTemplate<String, OrderPlacedEvent> needs to turn your Java object into bytes for the producer, and @KafkaListener needs to turn those bytes back into the right Java class on the consumer. Spring Kafka ships JsonSerializer and JsonDeserializer built on Jackson to handle this — but they have several sharp edges that break in real multi-service deployments. How Spring Kafka JSON Serialization Works flowchart LR subgraph Producer["Order Service"

The Cluster: Brokers and the Controller A Kafka cluster is a group of servers, each called a broker. Brokers store data and serve producer/consumer requests. One broker in the cluster acts as the controller — it manages partition leadership, handles broker joins and departures, and coordinates rebalancing. In KRaft mode (Kafka 3.3+, the default from Kafka 4.0), the controller is built into Kafka itself — no ZooKeeper needed. flowchart TB subgraph Cluster["

Why Learn the CLI First? Before writing any Spring code, understanding the Kafka CLI tools gives you the ability to: Verify your cluster is working correctly Inspect topics and partitions Debug consumer lag issues Replay messages from specific offsets Reset consumer groups during incident recovery All CLI tools are in Kafka’s bin/ directory. In Docker, run them with docker exec: docker exec kafka <tool> <args> kafka-topics.sh: Managing Topics Create a Topic docker exec kafka kafka-topics.

How @KafkaListener Works @KafkaListener is a Spring Kafka annotation that registers a method as a Kafka consumer. Under the hood, Spring Kafka creates a ConcurrentMessageListenerContainer — a managed thread pool that continuously polls the broker and dispatches records to your method. flowchart LR Broker["Kafka Broker"] subgraph Container["ConcurrentMessageListenerContainer"] T1["Poll Thread 1\n(Partition 0)"] T2["Poll Thread 2\n(Partition 1)"] T3["Poll Thread 3\n(Partition 2)"] end Method["@KafkaListener\nvoid onOrderPlaced(...)"] Broker -->|"fetch records"| T1 Broker -->|"fetch records"| T2 Broker -->|"

How a Spring Kafka Producer Works KafkaTemplate is the central Spring Kafka class for sending messages. It wraps the native Kafka KafkaProducer, manages serialization, and provides a Spring-friendly API for sending records. flowchart LR App["Your Service\n(OrderService)"] KT["KafkaTemplate\n(Spring Kafka)"] Buffer["Producer Buffer\n(RecordAccumulator)"] Sender["Sender Thread\n(NetworkClient)"] Broker["Kafka Broker\n(Leader Partition)"] App -->|"send(topic, key, value)"| KT KT -->|serialize + route| Buffer Buffer -->|batch when full\nor linger.ms elapsed| Sender Sender -->|ProduceRequest| Broker Broker -->|ProduceResponse| Sender Sender -->|callback| App The send is asynchronous by default — KafkaTemplate.

Kafka Streams vs @KafkaListener @KafkaListener is a consumer — it reads records and processes them one by one or in batches. Kafka Streams is a stream processing library — it builds a topology of transformations that runs continuously, with built-in state stores, windowed aggregations, and join operations. Aspect @KafkaListener Kafka Streams Processing model Consume and process Topology of operators Stateful processing Manual (external DB) Built-in state stores (RocksDB) Windowed aggregations Manual Native (time, session, hopping) Joins Manual KStream-KTable, KStream-KStream Fault tolerance Committed offsets Changelog topics + offsets Use when Imperative event handling Stream transformations and aggregations Maven Dependency <dependency> <groupId>org.

Why Transactions? At-least-once delivery means a record can be processed and produced more than once after a crash. For most applications, idempotent consumers handle this. But when you need a hard guarantee — either the produce happens and the offset commits, or neither does — you need Kafka transactions. Common scenarios: Consume → transform → produce (read from one topic, write to another) where partial completion is unacceptable Exactly-once aggregations in financial or billing systems Atomic multi-topic produce where records to multiple topics must all land or none land How Kafka Transactions Work sequenceDiagram participant Producer participant Broker participant Consumer Producer->>Broker: initTransactions() [registers transactional.