Kafka Partition & Throughput Sizer

Turn a target throughput and retention window into a partition count, cluster storage per broker, and a starter topics config for a new Kafka topic.

⚡ Workload Presets

Web click stream IoT telemetry (small) Debezium CDC App log shipping Metrics pipeline Video events Trade ticks

📈 Workload

Peak producer rate (msg/s)

Avg record size (bytes, incl. key + headers)

Peak-to-average burst factor

Consumer groups

🔧 Broker & Partition Assumptions

Single-partition write ceiling (MB/s)

Single-consumer read ceiling (MB/s)

Replication factor

Broker count

Retention

Retention unit

Compression ratio (1 = none, 0.3 = ~3×)

On-disk overhead (index, .snapshot, %)

📊 Sizing Result

–

Recommended partitions

–

Ingress throughput (MB/s)

–

Log per broker

–

Max consumer parallelism

–

Uncompressed bytes/s–

Compressed bytes/s–

Peak bytes/s (bursted)–

Retention seconds–

Log size before replication–

× replication (RF)–

× overhead–

Cluster on-disk footprint–

📜 Config Snippet

–

Paste into a Terraform confluentcloud_kafka_topic, Strimzi KafkaTopic, or feed straight to kafka-topics.sh --create. Retention is emitted in ms as Kafka expects.

🔄 Throughput vs. Partitions

Partitions	Ingress ceiling (MB/s)	Egress ceiling (MB/s, per group)	Headroom vs. peak

Notes

Rule of thumb. Confluent recommends at most 4,000 partitions per broker and 200,000 per cluster for classic ZooKeeper-mode Kafka; KRaft mode raises the ceiling but the per-partition metadata cost is the same. Under-partitioning caps producer throughput; over-partitioning inflates end-to-end latency and controller work.

Storage. A partition is a sequence of segment files (.log, .index, .timeindex). Index & snapshot files add ~5–15 % depending on segment size and retention. Log compaction can shrink footprint drastically for keyed topics, but this tool assumes a plain time-retained log.

Single-partition ceilings. The write ceiling depends on record size, disk IOPS, RF, and network. 10 MB/s is a widely-cited safe number for RF=3 on SSD; you can push a partition harder with idempotent large-batch producers, but a hot partition remains the single biggest latency risk.