Jürgen,
I have zk clusters in dynamic environment like Autoscalling groups and as
you know in ASG it is quite common for a instance to get terminate and new
one comes up right, so in that case if i rely on static config it will be
little bit hard to manage the cluster, i was thinking if we have RR DNS
name atleast i can update the DNS entry when new nodes comes up or old one
terminate. I have not played with dynamic config option yet but if that
solves the problem we see in dynamic environments i am good. And i am not
comparing with consul but just pointing out the existing example.
Alan,
Yes i am looking for the similar solution.
Thanks,
Ram
On Mon, Feb 11, 2019 at 6:52 AM Alan Scherger <alan.scher...@gmail.com>
wrote:
> Hey Jürgen,
>
> My intent was to simply suggest a more programmatic means for dynamic
> configuration. In particular, the detecting of seed nodes and their
> appropriate id numbers. One might imagine provisioning 3 nodes with tags
> like:
>
> zk_cluster=thebestcluster
> zk_myid={1,2,3}
>
> and then in the zk configuration we might have:
>
> discovery=ec2Tags
> discovery.ec2Tags.tagCluster=zk_cluster
> discovery.ec2Tags.tagMyid=zk_myid
>
> This would allow a little code to parse the tags out of ec2 and build the
> seed node configurations.
>
> Similarly we could build and maintain a custom auth provider that could use
> the AWS Certificate Manager Private CA APIs or Hashicorp Vault PKI APIs to
> automatically create and fetch the appropriate certificates and
> configurations.
>
> To your point, the security of introducing autoconfiguration of settings
> like these might not be appropriate for all folks or installations, but
> environments where things like instance level IAM exist help mitigate some
> risk assuming the proper access controls have been put in place. <Redacted
> rant :) >
>
> I believe it's the lack of autoconfiguration in Zookeeper that has led to
> the creation of tools like Exhibitor or other tools that have never been
> open sourced for one reason or another. The introduction of Dynamic
> Reconfiguration is quite great, but the 'Re' part might imply we still have
> some initial work left to be done.
>
> I'm also not sure how a RR DNS record mechanism would satisfy the id
> assignment requirement since typically the value of such a record is that
> the results never return in a guaranteed order. Historically, I've seen
> tools, Netflix's Eureka, over come such a challenge by use of TXT records
> instead. One might argue an SRV record with appropriate priority values
> could also ensure the ordering. However, personally none of this is
> particularly my cup of tea, but I do enjoy coming across and existence of
> such systems.
>
> Hopefully this helps? I'm certainly not trying to advocate for busy work,
> extensive feature design, or russle any jimmies.
>
> Alan Scherger
>
>
>
> On Mon, Feb 11, 2019, 12:43 AM Jürgen Wagner (DVT)
> <juergen.wag...@devoteam.com.invalid wrote:
>
> > ...and come to think of it, there is another question. Cloud instance
> > tags are simply labels. There is no real semantics associated or
> > mandated by tags.
> >
> > In particular, there is no guarantee that a Zk instance is running on,
> > e.g., an EC2 instance labelled as "Zookeeper". Tags don't make services.
> >
> > If you want to use auto-scaling to create more Zk nodes and reconfigure
> > an existing cluster, the cluster will take care of discovering its
> > members, so only clients would be affected by the changes. They, however
> > could start with a well-known set of Zk nodes (e.g., the initial three),
> > inquire about the actual configuration, and subscribe to configuration
> > changes. There is no need for a tag- or DNS-based grouping this way.
> >
> > If you wanted to say, "hey, all of you Zk instances in my VPC, form a
> > cluster right now", you could do this indeed with tagging to seed the
> > server list initially. However, keep in mind that Zk is often a
> > security-relevant component and you don't really want ANY new Zk server
> > out there to be able to join your precious cluster - only the ones you
> > know about already.
> >
> > The fact that Consul may support something like it, doesn't mean it also
> > makes sense for Zookeeper. Consul and Zookeeper protocols and
> > architectures are quite different.
> >
> > I still don't understand what the precise requirement is that lead to
> > this question.
> >
> > I hope you'll enlighten me :-)
> >
> > Cheers,
> >
> > --Jürgen
> >
> >
> > On 11.02.2019 01:20, rammohan ganapavarapu wrote:
> >
> > > Should I open a feature request? For both cloud auto discovery and use
> > DNS
> > > end point to form a quorum.
> > >
> > > On Sun, Feb 10, 2019, 3:56 PM Alan Scherger <alan.scher...@gmail.com>
> > wrote:
> > >
> > >> We might look at something like this:
> > https://github.com/hekate-io/hekate
> > >> for inspiration (or adoption). In the Golang community Hashicorp has
> > built
> > >> something similar: https://github.com/hashicorp/go-discover -- this
> > >> problem
> > >> set itself probably warrants a multilingual Apache project to help
> drive
> > >> some standards and interoperability.
> > >>
> > >> Alan
> > >>
> > >> On Sun, Feb 10, 2019, 5:42 PM rammohan ganapavarapu <
> > >> rammohanga...@gmail.com
> > >> wrote:
> > >>
> > >>> Clod providers have api to query instance IP based in tags, actually
> > >> consul
> > >>> is doing that to form a cluster.
> > >>>
> > >>> On Sun, Feb 10, 2019, 11:40 AM Andor Molnar
> <an...@cloudera.com.invalid
> > >
> > >>> wrote:
> > >>>
> > >>>> Hi Ram!
> > >>>>
> > >>>> What exactly do you mean by "auto-discovery on cloud instance tags"?
> > >>>> Is there a standard way of doing that?
> > >>>>
> > >>>> Regards,
> > >>>> Andor
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Sat, Feb 9, 2019 at 4:07 PM Norbert Kalmar
> > >>> <nkal...@cloudera.com.invalid
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Ram,
> > >>>>>
> > >>>>> Unfortunately ZK does not support RR DNS name.
> > >>>>> As for plans on discovery based on cloud tags, I am not aware of
> any
> > >>>> plans.
> > >>>>> You can create a jira for it if you'd like, but I can't tell you
> when
> > >>>> that
> > >>>>> would make it into a release.
> > >>>>>
> > >>>>> Regards,
> > >>>>> Norbert
> > >>>>>
> > >>>>> On Fri, Feb 8, 2019 at 11:53 PM rammohan ganapavarapu <
> > >>>>> rammohanga...@gmail.com> wrote:
> > >>>>>
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> Does zookeper support RR DNS name in the config instead of giving
> > >>> each
> > >>>>>> server name/ip like what consul does to join the cluster?
> > >>>>>>
> > >>>>>>
> > >>>>>> server.1=server1
> > >>>>>> server.2=server2
> > >>>>>> server.3=server3
> > >>>>>>
> > >>>>>> vs
> > >>>>>> server=example.com
> > >>>>>> where example.com is RR of server1, server2 and server3
> > >>>>>>
> > >>>>>> And does any one know if zk team has any plans to add cloud
> > >>>> autodiscovery
> > >>>>>> based on cloud instance tags?
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Ram
> > >>>>>>
> >
>
