@Yifan Ying Why not add more brokers in your cluster? That will not increase the partitions. Does increasing the number of brokers cause you any problem? How many brokers do you have in the cluster already?
On Wed, Dec 7, 2016 at 12:35 AM, Yifan Ying <nafan...@gmail.com> wrote: > Thanks Asaf, Aseem. > > Assigning topics to only a specific set of brokers will probably cause > uneven traffic and it won't prevent topics to be re-assigned to other > brokers when brokers fail. > > Like I said, the original cluster is close to out of resources. I remember > there's some limit on # of partitions that each Kafka cluster can have. > Netflix recommends to keep it below 10k to improve availability and reduce > latency, > http://techblog.netflix.com/2016/04/kafka-inside-keystone-pipeline.html. > Jun Rao also wrote a blog( > https://www.confluent.io/blog/how-to-choose-the-number-of- > topicspartitions-in-a-kafka-cluster/) > about how too many partitions could hurt availability and latency. That's > why we want to create another cluster instead of expanding the current one. > I know a lot of companies are maintaining multiple clusters, and I'm > curious how people are doing topic discovery. > > > > On Tue, Dec 6, 2016 at 4:04 AM, Aseem Bansal <asmbans...@gmail.com> wrote: > > > What configurations allow you to assign topics to specific brokers? > > > > I can see https://kafka.apache.org/documentation#basic_ops_automigrate. > > This should allow you to move things around but does that keep anything > > from being re-assigned to the old ones? > > > > On Tue, Dec 6, 2016 at 5:25 PM, Asaf Mesika <asaf.mes...@gmail.com> > wrote: > > > > > Why not re-use same cluster? You can assign topics to be live only > > within a > > > specific set of brokers. Thus you have one "bus" for messages, > > simplifying > > > your applications code and configurations > > > > > > On Mon, Dec 5, 2016 at 9:43 PM Yifan Ying <nafan...@gmail.com> wrote: > > > > > > > Hi, > > > > > > > > Initially, we have only one Kafka cluster shared across all teams. > But > > > now > > > > this cluster is very close to out of resources (disk space, # of > > > > partitions, etc.). So we are considering adding another Kafka > cluster. > > > But > > > > what's the best practice of topic discovery, so that applications > know > > > > which cluster their topics live? We have been using Zookeeper for > > service > > > > discovery, maybe it's also good for this purpose? > > > > > > > > Thanks > > > > > > > > -- > > > > Yifan > > > > > > > > > > > > > -- > Yifan >
