Thanks Asaf, Aseem. Assigning topics to only a specific set of brokers will probably cause uneven traffic and it won't prevent topics to be re-assigned to other brokers when brokers fail.
Like I said, the original cluster is close to out of resources. I remember there's some limit on # of partitions that each Kafka cluster can have. Netflix recommends to keep it below 10k to improve availability and reduce latency, http://techblog.netflix.com/2016/04/kafka-inside-keystone-pipeline.html. Jun Rao also wrote a blog( https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) about how too many partitions could hurt availability and latency. That's why we want to create another cluster instead of expanding the current one. I know a lot of companies are maintaining multiple clusters, and I'm curious how people are doing topic discovery. On Tue, Dec 6, 2016 at 4:04 AM, Aseem Bansal <asmbans...@gmail.com> wrote: > What configurations allow you to assign topics to specific brokers? > > I can see https://kafka.apache.org/documentation#basic_ops_automigrate. > This should allow you to move things around but does that keep anything > from being re-assigned to the old ones? > > On Tue, Dec 6, 2016 at 5:25 PM, Asaf Mesika <asaf.mes...@gmail.com> wrote: > > > Why not re-use same cluster? You can assign topics to be live only > within a > > specific set of brokers. Thus you have one "bus" for messages, > simplifying > > your applications code and configurations > > > > On Mon, Dec 5, 2016 at 9:43 PM Yifan Ying <nafan...@gmail.com> wrote: > > > > > Hi, > > > > > > Initially, we have only one Kafka cluster shared across all teams. But > > now > > > this cluster is very close to out of resources (disk space, # of > > > partitions, etc.). So we are considering adding another Kafka cluster. > > But > > > what's the best practice of topic discovery, so that applications know > > > which cluster their topics live? We have been using Zookeeper for > service > > > discovery, maybe it's also good for this purpose? > > > > > > Thanks > > > > > > -- > > > Yifan > > > > > > -- Yifan
