MG>Hopefully quick response > From: [email protected] > Date: Mon, 19 Sep 2016 11:19:00 -0700 > Subject: Consul / Zookeeper [was Re: any update on this?] > To: [email protected] > CC: [email protected] > > [+ dev list] > > I have not worked on KAFKA-1793 directly, but I believe most of the > work so far has been in removing all zookeeper dependencies from > clients. The two main areas for that are (1) consumer rebalancing, and MG>Which classes/interfaces handle consumer rebalance ? > (2) administrative commands. MG>Which classes/interfaces handle administrative commands ? > > 1) Consumer rebalancing APIs were added to the broker in 0.9. The "new > consumer" uses these apis and does not connect directly to zookeeper > to manage group leadership and rebalancing. So my understanding is > that this work is complete. MG>the TODO is to document what these classes are ..who is the tech lead on 0.9 rebalancing API? > > 2) Admin commands are being converted to direct API calls instead of > direct zookeeper access with KIP-4. A small part of this project was > released in 0.10.0.0 and are open PRs for additional chunks that may > make it into 0.10.1.0 . If you are interested in helping or getting > involved, you can follow the KIP-4 discussions on the dev mailing > list. MG>URL for the KIP-4 wiki/blog ? > > When the client issues are completed I think the next step will be to > refactor the broker's zookeeper access (zkUtils) into an abstract > interface that could potentially be provided by consul or etcd. With > an interface in place, it should be possible to write an alternate > implementation of that interface for consul.
MG>this is how maven implements wagon provider..you can use existing implementations of wagon interfaceMG>or of course create your own..anyone taking the lead on this so far? > > Hope this helps, MG>Thanks for taking the time to think consul/zookeeper integration through > -Dana MG>looks like we have work to do..I have a few cycles available starting Sunday..so I would definitely like to "lend a hand" > On Mon, Sep 19, 2016 at 6:31 AM, Martin Gainty <[email protected]> wrote: > > Jens/Kant > > not aware of any shortfall with zookeeper so perhaps you can suggest > > advantages for Consul vs Zookeeper? > > Maven (I am building, testing and running kafka internally with maven) > > implements wagon-providers for URLConnection vs HttpURLConnection > > wagonshttps://maven.apache.org/guides/mini/guide-wagon-providers.html > > Thinking a network_provider should work for integrating external network > > provider. how would you architect this integration? > > > > would a configurable network-provider such as maven-wagon-provider work for > > kafka?Martin > > > >> From: [email protected] > >> To: [email protected] > >> Subject: Re: any update on this? > >> Date: Mon, 19 Sep 2016 09:41:10 +0000 > >> > >> Yes ofcourse the goal shouldn't be moving towards consul. It should just be > >> flexible enough for users to pick any distributed coordinated system. > >> > >> > >> > >> > >> > >> > >> On Mon, Sep 19, 2016 2:23 AM, Jens Rantil [email protected] > >> wrote: > >> I think I read somewhere that the long-term goal is to make Kafka > >> > >> independent of Zookeeper alltogether. Maybe not worth spending time on > >> > >> migrating to Consul in that case. > >> > >> > >> > >> > >> Cheers, > >> > >> Jens > >> > >> > >> > >> > >> On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain <[email protected]> > >> > >> wrote: > >> > >> > >> > >> > >> > +2 watching. > >> > >> > > >> > >> > On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <[email protected]> wrote: > >> > >> > > >> > >> > > https://issues.apache.org/jira/browse/KAFKA-1793 > >> > >> > > It would be great to use Consul instead of Zookeeper for Kafka and I > >> > >> > think > >> > >> > > it > >> > >> > > would benefit Kafka a lot from the exponentially growing consul > >> > >> > community. > >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > -- > >> > >> > > >> > >> > > >> > >> > Jennifer Fountain > >> > >> > DevOPS > >> > >> > > >> > >> -- > >> > >> > >> > >> > >> Jens Rantil > >> > >> Backend Developer @ Tink > >> > >> > >> > >> > >> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden > >> > >> For urgent matters you can reach me at +46-708-84 18 32. > >
