Luigi, I strongly urge you to consider a 5 node ZK deployment. I've always done that in the past for resiliency during maintenance. In a 3 node cluster, you can only tolerate one "failure", so if you bring one node down for maintenance and another node crashes during said maintenance, your ZK cluster is down. All the deployments I've had were 5 nodes of ZK and 5 nodes of Kafka.
- Pradeep On Thu, Jul 20, 2017 at 9:12 AM, Luigi Tagliamonte < luigi.tagliamont...@gmail.com> wrote: > Yes Andrey, > you can use an ENI without EIP on AWS if you only want a private address. > > After some consideration, I think that growing the zookeeper cluster more > than 3 nodes is really unlikely so I think that I will attach 3 ENI to 3 > servers in autoscaling and I will configure Kafka in using this 3 IPs. > In this way I can get rid of the additional ELB/Haproxy layer, if I will > ever need to grow the zk ensemble I will re-engineering the solution. > > I'm wondering if reusing an old IP on a brand new zk node will create > issues in the ensemble. > Is anybody here aware of possible drawbacks? > > On Wed, Jul 19, 2017 at 11:58 PM, Andrey Dyachkov < > andrey.dyach...@gmail.com > > wrote: > > > The problem with EIP it is a public ip. > > Another option is to have the secondary interface attached to the > instance > > on start(or a bit later) with the private static ip, but we are > > investigating the possibility. > > On Wed 19. Jul 2017 at 23:38, Luigi Tagliamonte < > > luigi.tagliamont...@gmail.com> wrote: > > > > > Hello Andrey, > > > I see that the ELB is not going to help directly with the bug, but > > > introduces a nice layer that makes zookeeper DNS management easier. > > > Introducing and ELB I don't have to deal with keep DNS in sync for all > > the > > > servers in the zk ensemble. > > > For the moment I can use an HAproxy with EIP and when the bug is > solved I > > > can move to ELB. > > > What do you think about it? > > > Regards > > > L. > > > > > > On Wed, Jul 19, 2017 at 2:16 PM, Andrey Dyachkov < > > > andrey.dyach...@gmail.com> > > > wrote: > > > > > > > Hi, > > > > I have just posted almost the same question in dev list. > > > > Zookeeper client resolves address only once, on start, introducing > ELB > > > > won't really help here (ELBs can be replaced, which involved ip > > change), > > > > but I am eager to know if there is a solution for that. > > > > > > > > On Wed, 19 Jul 2017 at 23:08 Luigi Tagliamonte < > > > > luigi.tagliamont...@gmail.com> wrote: > > > > > > > > > Hello, Users! > > > > > I'm designing a Kafka deployment on AWS and it's my first time > > working > > > > with > > > > > Kafka and Zookeeper so I've collected a lot of info so far but also > > > some > > > > > questions that I would love to submit to a much expert audience > like > > > you. > > > > > > > > > > I have been experimenting with exhibitor and zookeeper in auto > > scaling > > > > > group and the exhibitor orchestration seems to work so far. > > > > > > > > > > I was trying to find a way to configure zookeeper servers in Kafka > > conf > > > > and > > > > > do not have to reconfigure them in case a zookeeper node needs to > be > > > > > replaced/dies, so i taught of course of using DNS but then I read > > that > > > > > zkclient library used by Kafka has this bug: > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2184. > > > > > > > > > > So I'm now thinking about using an ELB in front of the zookeeper > > > cluster. > > > > > Teorically on how zookeeper client should work there should be no > > > problem > > > > > but I'm wondering if any of you used that and how is the outcome? > > > > > > > > > -- > > > > > > > > With great enthusiasm, > > > > Andrey > > > > > > > > > -- > > > > With great enthusiasm, > > Andrey > > >
