Thanks for your comments Alex. The KIP proposes using two configurations max.partitions and max.broker.partitions. It does not enforce their use. The default values are pretty large (INT MAX), therefore should be non-intrusive.
In multi-tenant environments and in partition assignment and rebalancing, the admin could (a) use the default values which would yield similar behavior to now, (b) set very high values that they know is sufficient, (c) dynamically re-adjust the values should the business requirements change. Note that the two configurations are cluster-wide, so they can be updated without restarting the brokers. The quota system in Kafka seems to be geared towards limiting traffic for specific clients or users, or in the case of replication, to leaders and followers. The quota configuration itself is very similar to the one introduced in this KIP i.e. just a few configuration options to specify the quota. The main difference is that the quota system is far more heavy-weight because it needs to be applied to traffic that is flowing in/out constantly. Whereas in this KIP, we want to limit number of partition replicas, which gets modified rarely by comparison in a typical cluster. Hope this addresses your comments. On Thu, Apr 9, 2020 at 12:53 PM Alexandre Dupriez < alexandre.dupr...@gmail.com> wrote: > Hi Gokul, > > Thanks for the KIP. > > From what I understand, the objective of the new configuration is to > protect a cluster from an overload driven by an excessive number of > partitions independently from the load handled on the partitions > themselves. As such, the approach uncouples the data-path load from > the number of unit of distributions of throughput and intends to avoid > the degradation of performance exhibited in the test results provided > with the KIP by setting an upper-bound on that number. > > Couple of comments: > > 900. Multi-tenancy - one concern I would have with a cluster and > broker-level configuration is that it is possible for a user to > consume a large proportions of the allocatable partitions within the > configured limit, leaving other users with not enough partitions to > satisfy their requirements. > > 901. Quotas - an approach in Apache Kafka to set-up an upper-bound on > resource consumptions is via client/user quotas. Could this framework > be leveraged to add this limit? > > 902. Partition assignment - one potential problem with the new > repartitioning scheme is that if a subset of brokers have reached > their number of assignable partitions, yet their data path is > under-loaded, new topics and/or partitions will be assigned > exclusively to other brokers, which could increase the likelihood of > data-path load imbalance. Fundamentally, the isolation of the > constraint on the number of partitions from the data-path throughput > can have conflicting requirements. > > 903. Rebalancing - as a corollary to 902, external tools used to > balance ingress throughput may adopt an incremental approach in > partition re-assignment to redistribute load, and could hit the limit > on the number of partitions on a broker when a (too) conservative > limit is used, thereby over-constraining the objective function and > reducing the migration path. > > Thanks, > Alexandre > > Le jeu. 9 avr. 2020 à 00:19, Gokul Ramanan Subramanian > <gokul24...@gmail.com> a écrit : > > > > Hi. Requesting you to take a look at this KIP and provide feedback. > > > > Thanks. Regards. > > > > On Wed, Apr 1, 2020 at 4:28 PM Gokul Ramanan Subramanian < > > gokul24...@gmail.com> wrote: > > > > > Hi. > > > > > > I have opened KIP-578, intended to provide a mechanism to limit the > number > > > of partitions in a Kafka cluster. Kindly provide feedback on the KIP > which > > > you can find at > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions > > > > > > I want to specially thank Stanislav Kozlovski who helped in formulating > > > some aspects of the KIP. > > > > > > Many thanks, > > > > > > Gokul. > > > >
