Hi Boyang, The Kafka clusters we manage are multi-tenant clusters hosting anywhere from hundreds to a few thousand different workloads on any given cluster.
For our setup, we have noticed that the breaking limit wrt partition count is around 10k partitions per broker. Beyond this point, we start seeing significant replication slowness, election slowness, issues around too many files opened etc The type of workloads on our clusters that would benefit from the proposal outlined in this KIP are *Bursty workloads such as workloads that flood the topic once an hour and need to be processed quickly within a strict time window *Workloads that are using topics as simple queues (stateless and don’t care about ordering within a partition) *Stream processing workloads where parallelism is driven by the number of input topic partitions Currently, we are over provisioning partitions to efficiently serve these workloads which results in significant under-utilization of the respective clusters. Additionally, we are also seeing quite a few workloads that are relying on the partition level ordering guarantees today and are filtering out the keys they don’t care about on the client side. These workloads would benefit from the key level ordering proposed in KIP-X and result in much simpler application logic for clients. Let me know if this helps and if you have any further questions Rangan From: dev@kafka.apache.org At: 02/21/20 15:45:49To: dev@kafka.apache.org Cc: Rangan Prabhakaran (BLOOMBERG/ 919 3RD A ) , bche...@outlook.com Subject: Re: Discussion thread for KIP-X Hey Rangan, thanks for the interest! In fact we are still in the design phase, and need more supporting use cases that requires a higher scaling factor than number of partitions. It would be good if you could share some of your needed use case when the unit time of processing one record is the bottleneck, or some cost wise concern of over-partitioning. Boyang On Fri, Feb 21, 2020 at 10:44 AM Guozhang Wang <wangg...@gmail.com> wrote: > cc @Boyang Chen <bche...@outlook.com> who authored this draft. > > > Guozhang > > On Fri, Feb 21, 2020 at 10:29 AM Rangan Prabhakaran (BLOOMBERG/ 919 3RD A) > < > kprabhaka...@bloomberg.net> wrote: > > > Hi, > > A few of us have been following KIP-X. We are interested in the roadmap / > > plan there and would like to contribute towards the same. > > > > What are the next steps to discuss / iterate on this KIP ? Currently, its > > in draft state and there does not seem to be a discussion thread attached > > yet. > > > > KIP - > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-X%3A+Introduce+a+cooperati ve+consumer+processing+semantic > > > > Thanks > > Rangan > > > > -- > -- Guozhang >
