Hi Aseem, the concern is to create too many partitions in total in one cluster no matter how many brokers I have in this cluster. I think the two articles that I mentioned explain why too many partitions in one cluster could cause issues.
On Tue, Dec 6, 2016 at 12:08 PM, Aseem Bansal <asmbans...@gmail.com> wrote: > @Yifan Ying Why not add more brokers in your cluster? That will not > increase the partitions. Does increasing the number of brokers cause you > any problem? How many brokers do you have in the cluster already? > > On Wed, Dec 7, 2016 at 12:35 AM, Yifan Ying <nafan...@gmail.com> wrote: > > > Thanks Asaf, Aseem. > > > > Assigning topics to only a specific set of brokers will probably cause > > uneven traffic and it won't prevent topics to be re-assigned to other > > brokers when brokers fail. > > > > Like I said, the original cluster is close to out of resources. I > remember > > there's some limit on # of partitions that each Kafka cluster can have. > > Netflix recommends to keep it below 10k to improve availability and > reduce > > latency, > > http://techblog.netflix.com/2016/04/kafka-inside-keystone-pipeline.html. > > Jun Rao also wrote a blog( > > https://www.confluent.io/blog/how-to-choose-the-number-of- > > topicspartitions-in-a-kafka-cluster/) > > about how too many partitions could hurt availability and latency. That's > > why we want to create another cluster instead of expanding the current > one. > > I know a lot of companies are maintaining multiple clusters, and I'm > > curious how people are doing topic discovery. > > > > > > > > On Tue, Dec 6, 2016 at 4:04 AM, Aseem Bansal <asmbans...@gmail.com> > wrote: > > > > > What configurations allow you to assign topics to specific brokers? > > > > > > I can see https://kafka.apache.org/documentation#basic_ops_automigrate > . > > > This should allow you to move things around but does that keep anything > > > from being re-assigned to the old ones? > > > > > > On Tue, Dec 6, 2016 at 5:25 PM, Asaf Mesika <asaf.mes...@gmail.com> > > wrote: > > > > > > > Why not re-use same cluster? You can assign topics to be live only > > > within a > > > > specific set of brokers. Thus you have one "bus" for messages, > > > simplifying > > > > your applications code and configurations > > > > > > > > On Mon, Dec 5, 2016 at 9:43 PM Yifan Ying <nafan...@gmail.com> > wrote: > > > > > > > > > Hi, > > > > > > > > > > Initially, we have only one Kafka cluster shared across all teams. > > But > > > > now > > > > > this cluster is very close to out of resources (disk space, # of > > > > > partitions, etc.). So we are considering adding another Kafka > > cluster. > > > > But > > > > > what's the best practice of topic discovery, so that applications > > know > > > > > which cluster their topics live? We have been using Zookeeper for > > > service > > > > > discovery, maybe it's also good for this purpose? > > > > > > > > > > Thanks > > > > > > > > > > -- > > > > > Yifan > > > > > > > > > > > > > > > > > > > > -- > > Yifan > > > -- Yifan