Hi Aseem, the concern is to create too many partitions in total in one
cluster no matter how many brokers I have in this cluster. I think the two
articles that I mentioned explain why too many partitions in one cluster
could cause issues.

On Tue, Dec 6, 2016 at 12:08 PM, Aseem Bansal <asmbans...@gmail.com> wrote:

> @Yifan Ying Why not add more brokers in your cluster? That will not
> increase the partitions. Does increasing the number of brokers cause you
> any problem? How many brokers do you have in the cluster already?
>
> On Wed, Dec 7, 2016 at 12:35 AM, Yifan Ying <nafan...@gmail.com> wrote:
>
> > Thanks Asaf, Aseem.
> >
> > Assigning topics to only a specific set of brokers will probably cause
> > uneven traffic and it won't prevent topics to be re-assigned to other
> > brokers when brokers fail.
> >
> > Like I said, the original cluster is close to out of resources. I
> remember
> > there's some limit on # of partitions that each Kafka cluster can have.
> > Netflix recommends to keep it below 10k to improve availability and
> reduce
> > latency,
> > http://techblog.netflix.com/2016/04/kafka-inside-keystone-pipeline.html.
> > Jun Rao also wrote a blog(
> > https://www.confluent.io/blog/how-to-choose-the-number-of-
> > topicspartitions-in-a-kafka-cluster/)
> > about how too many partitions could hurt availability and latency. That's
> > why we want to create another cluster instead of expanding the current
> one.
> > I know a lot of companies are maintaining multiple clusters, and I'm
> > curious how people are doing topic discovery.
> >
> >
> >
> > On Tue, Dec 6, 2016 at 4:04 AM, Aseem Bansal <asmbans...@gmail.com>
> wrote:
> >
> > > What configurations allow you to assign topics to specific brokers?
> > >
> > > I can see https://kafka.apache.org/documentation#basic_ops_automigrate
> .
> > > This should allow you to move things around but does that keep anything
> > > from being re-assigned to the old ones?
> > >
> > > On Tue, Dec 6, 2016 at 5:25 PM, Asaf Mesika <asaf.mes...@gmail.com>
> > wrote:
> > >
> > > > Why not re-use same cluster? You can assign topics to be live only
> > > within a
> > > > specific set of brokers. Thus you have one "bus" for messages,
> > > simplifying
> > > > your applications code and configurations
> > > >
> > > > On Mon, Dec 5, 2016 at 9:43 PM Yifan Ying <nafan...@gmail.com>
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Initially, we have only one Kafka cluster shared across all teams.
> > But
> > > > now
> > > > > this cluster is very close to out of resources (disk space, # of
> > > > > partitions, etc.). So we are considering adding another Kafka
> > cluster.
> > > > But
> > > > > what's the best practice of topic discovery, so that applications
> > know
> > > > > which cluster their topics live? We have been using Zookeeper for
> > > service
> > > > > discovery, maybe it's also good for this purpose?
> > > > >
> > > > > Thanks
> > > > >
> > > > > --
> > > > > Yifan
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Yifan
> >
>



-- 
Yifan

Reply via email to