Why ZooKeeper Had to Go For the first decade of Kafka’s existence, every Kafka cluster required an Apache ZooKeeper cluster to manage metadata: controller election, topic configurations, partition leadership, access control lists, and consumer group state. This created real problems: flowchart TB subgraph OldArchitecture["Old Architecture: Kafka + ZooKeeper"] subgraph ZK["ZooKeeper Cluster (3+ nodes)"] Z1["ZK Node 1"] Z2["ZK Node 2"] Z3["ZK Node 3"] end subgraph Kafka["Kafka Cluster"] K1["Broker 1\n(Controller)"] K2["Broker 2"] K3["

Prerequisites Docker Desktop installed and running docker compose v2 (bundled with Docker Desktop 4.x+) Ports 9092, 9093, 9094 free on your machine All articles in this series assume a running local Kafka cluster. Start with the single-broker setup for articles 1–6, then switch to the 3-broker cluster when we cover replication and fault tolerance. Single-Broker Cluster (Development) This is the simplest setup — one Kafka node running in combined mode (broker + controller).