Re: Re: Topic discovery when supporting multiple kafka clusters

Hi Brian, We have 5 brokers and ~80 topics. And the total # of partitions is around 7k partitions if not including replicas (So it's close to the limit that Netflix recommends). Most topics have RF as 2. CPU is only around 25% usage. The average consumers for each topic should be around 3-4. Our

Re: Re: Topic discovery when supporting multiple kafka clusters

You didn't mention anything about your current configuration, just that you are 'out of resources'. Perhaps you misunderstand how to size your partitions per topic, and how partition allocation works. If your brokers are maxed on cpu, and you double the number of brokers but keep the replica

Re: Topic discovery when supporting multiple kafka clusters

Hi Aseem, the concern is to create too many partitions in total in one cluster no matter how many brokers I have in this cluster. I think the two articles that I mentioned explain why too many partitions in one cluster could cause issues. On Tue, Dec 6, 2016 at 12:08 PM, Aseem Bansal

Re: Topic discovery when supporting multiple kafka clusters

@Yifan Ying Why not add more brokers in your cluster? That will not increase the partitions. Does increasing the number of brokers cause you any problem? How many brokers do you have in the cluster already? On Wed, Dec 7, 2016 at 12:35 AM, Yifan Ying wrote: > Thanks Asaf,

Re: Topic discovery when supporting multiple kafka clusters

Thanks Asaf, Aseem. Assigning topics to only a specific set of brokers will probably cause uneven traffic and it won't prevent topics to be re-assigned to other brokers when brokers fail. Like I said, the original cluster is close to out of resources. I remember there's some limit on # of

Re: Topic discovery when supporting multiple kafka clusters

What configurations allow you to assign topics to specific brokers? I can see https://kafka.apache.org/documentation#basic_ops_automigrate. This should allow you to move things around but does that keep anything from being re-assigned to the old ones? On Tue, Dec 6, 2016 at 5:25 PM, Asaf Mesika

Re: Topic discovery when supporting multiple kafka clusters

Why not re-use same cluster? You can assign topics to be live only within a specific set of brokers. Thus you have one "bus" for messages, simplifying your applications code and configurations On Mon, Dec 5, 2016 at 9:43 PM Yifan Ying wrote: > Hi, > > Initially, we have only