I have also seen slower replication across the cluster when partitions per
broker are abnormally high, even though the bytes/message throughput isn't
that high. Due to legacy reasons, we have a lot of partitions per broker,
with only a handful really hot and the others just barely trickling data,
or even completely unused. And I can clearly see that it takes a while for
the brokers to service those replication requests. There's just more
metadata management that the brokers have to do internally, even when the
extra partitions are unused. The limit appears to be on a per-broker basis,
I suspect if we added more brokers the cluster would be just fine.
On Dec 14, 2017 6:00 AM, "Todd Palino" wrote:
Not recommended. You’ll have timeout issues with the size of the controller
requests. Additionally, there appear to be problems with writing some nodes
in Zookeeper at high partition counts.
-Todd
On Thu, Dec 14, 2017 at 8:58 AM, Abhimanyu Nagrath <
abhimanyunagr...@gmail.com> wrote:
> Can I have 20k partition on a single kafka broker ?
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
