#nailedit On Tue, Apr 14, 2020 at 2:05 PM Jamie <jamied...@aol.co.uk> wrote:
> Hi All, > > There is a KIP which might be of interest to you: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120736982 - > it sounds like you want to blacklist brokers in DC3? > > Thanks, > > Jamie > > -----Original Message----- > From: Michael K. Edwards <m.k.edwa...@gmail.com> > To: dev@kafka.apache.org > Sent: Tue, 14 Apr 2020 20:50 > Subject: Re: Preferred Partition Leaders > > I have clients with a similar need relating to disaster recovery. (Three > replicas per partition within a data center / AWS AZ/region, fourth replica > elsewhere, ineligible to become the partition leader without manual > intervention.) > > On Tue, Apr 14, 2020 at 12:31 PM Łukasz Antoniak < > lukasz.anton...@gmail.com> > wrote: > > > Hi Everyone, > > > > Recently I came across Kafka setup where two data centers are close to > each > > other, but the company could not find a suitable place for the third one. > > As a result third DC is little further, lower network throughput, but > still > > within range of decent network latency, qualifying for stretch cluster. > Let > > us assume that client applications are being deployed only on two > "primary" > > DCs. My idea was to minimize network traffic between DC3 and other data > > centers (ideally only to replication). > > > > For Kafka consumer, we can configure rack-awareness, so that consumers > will > > read data from closest replica (replica.selector.class). > > Kafka producers have to send data to partition leaders. There is no way > to > > tell that we prefer replica leaders to be running in DC1 and DC2. Kafka > > will also try to evenly balance leaders across brokers > > (auto.leader.rebalance.enable). > > > > Does it sound like a good feature to make the choice of partition leaders > > pluggable? Basically, users would be given list of topic-partitions with > > ISRs and rack they are running, and could reshuffle them according to > > custom logic. > > > > Comments appreciated. > > > > Kind regards, > > Lukasz > > >
