@EmbeddedKafka has long been the standard for Kafka testing in Spring Boot. It works, but it runs a different Kafka implementation than production — one that can accept messages and serialize records in ways that a real Kafka broker would not. KafkaContainer runs the same Confluent Kafka image your production environment uses, with the same broker configuration, same partition behavior, and same consumer group offset management. This article shows how to test Kafka producers, consumers, and dead-letter topics with a real broker.

@SendTo — Chaining Listeners @SendTo on a @KafkaListener method automatically sends the return value to another Kafka topic. This is how you build event pipelines without manually calling KafkaTemplate.send() in your listener. flowchart LR T1["orders\n(OrderPlacedEvent)"] T2["orders-confirmed\n(OrderConfirmedEvent)"] T3["inventory-events\n(StockReservedEvent)"] T1 -->|"@KafkaListener\n@SendTo"| L1["confirmOrder()"] L1 --> T2 T2 -->|"@KafkaListener\n@SendTo"| L2["reserveStock()"] L2 --> T3 Basic @SendTo @KafkaListener(topics = "orders", groupId = "confirmation-service") @SendTo("orders-confirmed") public OrderConfirmedEvent onOrder(OrderPlacedEvent event) { // Return value is automatically sent to "orders-confirmed" return new OrderConfirmedEvent( event.

Why Avro and Schema Registry? JSON has no schema enforcement — a producer can change a field name and silently break every consumer. Avro + Schema Registry solves this: Avro gives you a compact binary format with a schema definition Schema Registry stores and versions schemas, enforces compatibility rules, and prevents breaking changes from reaching consumers flowchart LR subgraph Producer["Order Service"] E["OrderPlacedEvent"] -->|"KafkaAvroSerializer"| SR["Schema Registry\n(register/lookup schema)"] SR --> Bytes["[schema_id (4 bytes)] + [avro payload]"

Why @Bean Configuration? application.properties covers the common cases. But real applications need multiple listener factories — one for orders with manual acknowledgment and concurrency 3, another for analytics events with batch listening and different deserializers. @Bean configuration gives you a factory per use case, full IDE support, and the ability to wire in custom components like error handlers and interceptors. The Factory Relationship flowchart TD CF["ConsumerFactory\n(connection + deserialization config)"] LCF["ConcurrentKafkaListenerContainerFactory\n(container behaviour config)"

What Is a Consumer Group? A consumer group is a set of consumer instances that jointly consume a topic. Kafka assigns each partition to exactly one consumer within the group at a time. This is what enables parallel processing: multiple consumers in the same group read different partitions simultaneously. flowchart LR subgraph Topic["Topic: orders — 4 partitions"] P0["Partition 0"] P1["Partition 1"] P2["Partition 2"] P3["Partition 3"] end subgraph CG["Consumer Group: inventory-service"] C1["

Consumer Group Fundamentals A consumer group is how Kafka distributes work across multiple consumers. Each partition in a topic is assigned to exactly one consumer instance in the group at any given time. flowchart TB subgraph Topic["Topic: orders — 6 partitions"] P0["P0"] P1["P1"] P2["P2"] P3["P3"] P4["P4"] P5["P5"] end subgraph CG1["Group: inventory-service (3 instances)"] I1["Instance 1\nP0, P1"] I2["Instance 2\nP2, P3"] I3["Instance 3\nP4, P5"] end subgraph CG2["Group: notification-service (1 instance)"] N1["Instance 1\nP0,P1,P2,P3,P4,P5"] end P0 & P1 --> I1 P2 & P3 --> I2 P4 & P5 --> I3 P0 & P1 & P2 & P3 & P4 & P5 --> N1 Both groups receive all events — they are independent.

When to Write a Custom Serializer Spring Kafka ships JSON and Avro support. You need a custom serializer when: Your team uses Protobuf or MessagePack and wants native support You need a compact binary format for high-throughput topics (pricing ticks, sensor readings) You’re integrating with a legacy system that publishes a fixed binary protocol You want deterministic serialization for event deduplication or content-addressed storage The Serializer and Deserializer Interfaces // org.

What Is a Dead Letter Topic? When a record fails processing and retries are exhausted, you have two options: skip it (losing the data) or park it somewhere for inspection and reprocessing. A dead-letter topic (DLT) is that parking lot — a Kafka topic that holds records that could not be processed, enriched with error metadata headers. flowchart LR subgraph Main["orders topic"] R1["record: offset 42\n(bad data)"] end subgraph DLT["orders.DLT topic"] R2["

Why Dynamic Listeners? @KafkaListener is declared at compile time. Some scenarios require listeners created at runtime: Multi-tenant SaaS — each tenant onboards to their own topic; you can’t redeploy to add @KafkaListener for each new tenant Feature flags — enable or disable a listener without a deployment Plugin systems — modules register their own topic subscriptions when loaded Admin APIs — operators subscribe to new topics via a REST endpoint ConcurrentMessageListenerContainer The core building block is ConcurrentMessageListenerContainer — the same class @KafkaListener uses internally, but constructed and started manually:

What Happens When a Listener Throws? Without an error handler, an uncaught exception from @KafkaListener causes the container to log the error and retry the same record on the next poll — indefinitely. One bad record can block an entire partition forever. DefaultErrorHandler fixes this: it retries a configurable number of times with backoff, then calls a ConsumerRecordRecoverer (e.g. send to a dead-letter topic) and moves on. DefaultErrorHandler — The Modern API Spring Kafka 2.