Re: Recommended max number of topics (and data separation)

After brief search, I found KAFKA-6469 FYI Original message From: Andrey Falko <afa...@salesforce.com> Date: 2/1/18 5:28 PM (GMT-08:00) To: users@kafka.apache.org Subject: Re: Recommended max number of topics (and data separation) Indeed David, I confirmed that I can'

Re: Recommended max number of topics (and data separation)

Indeed David, I confirmed that I can't push my clusters to more than 72k topics with default zookeeper settings. Once I get to that quantity, leader election never happens for new topics. Additionally if I kill one of the brokers, all the topics don't get leaders re-elected and it is impossible to

Re: Recommended max number of topics (and data separation)

I used: -Djute.maxbuffer=50111000 and the gain I had is that I could increment number of topics from 70k to 100k :P 2018-01-30 23:25 GMT+01:00 Andrey Falko : > On Tue, Jan 30, 2018 at 1:38 PM, David Espinosa wrote: > > Hi Andrey, > > My topics are

Re: Recommended max number of topics (and data separation)

On Tue, Jan 30, 2018 at 1:38 PM, David Espinosa wrote: > Hi Andrey, > My topics are replicated with a replicated factor equals to the number of > nodes, 3 in this test. > Didn't know about the kip-227. > The problems I see at 70k topics coming from ZK are related to any >

Re: Recommended max number of topics (and data separation)

Hi Andrey, My topics are replicated with a replicated factor equals to the number of nodes, 3 in this test. Didn't know about the kip-227. The problems I see at 70k topics coming from ZK are related to any operation where ZK has to retrieve topics metadata. Just listing topics at 50K or 60k you

Re: Recommended max number of topics (and data separation)

On Sun, Jan 28, 2018 at 8:45 AM, David Espinosa wrote: > Hi Monty, > > I'm also planning to use a big amount of topics in Kafka, so recently I > made a test within a 3 nodes kafka cluster where I created 100k topics with > one partition. Sent 1M messages in total. Are your

Re: Recommended max number of topics (and data separation)

Hi Monty, I'm also planning to use a big amount of topics in Kafka, so recently I made a test within a 3 nodes kafka cluster where I created 100k topics with one partition. Sent 1M messages in total. These are my conclusions: - There is not any limitation on kafka regarding the number of

Recommended max number of topics (and data separation)

I'm designing a system and need some more clarity regarding Kafka's recommended limits on the number of topics and/or partitions. At a high level, our system would work like this: - A user creates a job X (X is a UUID). - The user uploads data for X to an input topic: X.in. - Workers process the